Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
bee0a66
Read Bend 2 code, and judge whether its laws state what their comment…
tauanbinato Sep 26, 2026
6c06ccf
Ask an undecided law what its comment adds, and leave units refused f…
tauanbinato Sep 27, 2026
7d31bfb
Ask an undecided law whether it checks particular inputs its comment …
tauanbinato Sep 27, 2026
08fbf45
Judge a law with the uncommented laws its comment heads, and keep sib…
tauanbinato Sep 27, 2026
50cdb52
Tune Bend 2 values and function questions from labels
tauanbinato Sep 27, 2026
a17964c
Keep copies between separate Bend 2 tests apart
tauanbinato Sep 27, 2026
b2a5c0e
Test that copies between Bend 2 golden tests are not paired
tauanbinato Sep 27, 2026
e1a5797
Describe Bend 2 support, the laws rule and the new skip reasons
tauanbinato Sep 27, 2026
e2c219e
Leave Bend 2 proofs out of function simplification and shared logic
tauanbinato Sep 27, 2026
8db84a2
Judge a Bend 2 program on a test path as a test
tauanbinato Sep 27, 2026
2cd264f
Take a law's comment from the block directly above it
tauanbinato Sep 27, 2026
d893345
Weigh a Bend 2 file's split only past 300 member lines
tauanbinato Sep 27, 2026
83a51e1
Bump the rule versions changed on this branch
tauanbinato Sep 27, 2026
73196bb
Describe this round's Bend 2 changes, and show a law finding that holds
tauanbinato Sep 27, 2026
22168d1
Tell Bend 2 files of proofs by their name or directory
tauanbinato Sep 27, 2026
10c818d
Compare Bend 2 proofs for copies again
tauanbinato Sep 27, 2026
2640bcc
Leave the values of Bend 2 benchmarks out
tauanbinato Sep 27, 2026
a7e24f9
Make a split of a Bend 2 file laid out in titled sections a note
tauanbinato Sep 27, 2026
ea70d9f
Make a split of a Bend 2 file at most a consider
tauanbinato Sep 27, 2026
5b18854
Describe the last Bend 2 changes and their measurement
tauanbinato Sep 27, 2026
95de0e6
Split law_predicates into its three steps
tauanbinato Sep 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,15 @@ Notable changes to JevGate. Versions follow [Semantic Versioning](https://semver

## [Unreleased]

Bend 2 ([bendlang/bend](https://github.com/bendlang/bend) 2.0.x) is a supported language, with a rule for its laws. Every review and consider on Bend code was labeled by hand from the code, a debatable one counting as not right. On the Bend repository and 40 community projects used for tuning, 58% of reviews and 43% of considers were right (62% and 44% for the default rules, without the opt-in security and documentation groups). On 23 community projects never used for tuning, 53% of reviews were right before one change made from their labels (a split of a Bend 2 file is at most a consider) and 70% with it (74% for the default rules), and about 27% of considers, from a sample of 150 of 427; hardcoded values were the weakest rule there (26% right), and law findings were right 15 times in 23. Every request to the other languages' code is unchanged on the 117 corpus projects.

- Bend 2: `.bend` files are parsed with tree-sitter-bend2. Defs, types and laws are units, named with their dots (`List.map`), and a call through an import alias (`Sort.sort` beside `import ./main.bend as Sort`) reaches the def it names; imports link files by their paths, and `import Base` links Base in the Bend repository. Jev is told Bend 2's notation beside each file's code, since a model may know Bend 1 or no Bend at all. A test is a program whose file ends in the `#|` lines its run must print, or on a test path any program that defines `main`, as bendc's `tests/X.bend` beside its `X.out`, judged whole; the `#|` lines are no comment. Proofs (a def that fills a claim or states an equality, or any def of a file of proofs: a `PROOF.bend`, a file named after what it proves such as `padding_proof.bend`, or one under a `proof` or `proofs` directory) and type-level defs are told from code: proofs are not asked to be split (the 16 such findings were wrong), neither is asked about hardcoded values, the laws of a file of proofs are lemmas and not judged, copies of proof steps are compared like other code (32 of 41 were right in a library of proofs), and only defs that perform effects (`IO`, a `do IO` block, a foreign import, a def filling an `IO` law) or join text with `++` are asked the security questions. `LAWS.bend` and `PROOF.bend` are not asked to be split, nor a Bend 2 file of fewer than 300 member lines (`min_bend_file_lines` in the decision policy; the 16 file-organization findings below it were wrong, and the 13 right ones above it), and a split of a Bend 2 file is at most a consider (14 of 43 such reviews were right), a note when its author ruled the file off into titled sections (`# ----`; 7 of 43 such findings were right, against 10 of 17 elsewhere); a benchmark's values are its workload and are not asked about (70 of 82 such findings were wrong), a `base.bend` copied from Bend's Base library is vendored, and copies between sibling benchmark programs or two tests pinned to their output are not compared.
- Bend 1 files, a different language that shares the `.bend` extension, are skipped with their own reason before parsing: 497 of the 525 files of Bend 1's repository (the rest are fragments that do not parse either), and none of the 3,713 Bend 2 files of the Bend repository and 40 community projects.
- New rule `tests/laws` (on by default, judged in application code): a Bend 2 law is the part of a specification the compiler checks and its comment the part a person reads, so each claim that quantifies over its inputs and has a comment is asked whether the comment claims more than the law states, with the law read in words (`for every page: String, there is some head: String such that …`), the laws right after it that its comment also heads (its comment is the block directly above it, not the paragraph that opens its section), the file's header comment and the definitions it names. An undecided law is asked what its comment adds and whether the law checks only particular inputs its comment generalizes (the 4 at 0.65 or more were right). It finds laws such as bend-json's `remove_sound`, which checks a one-entry object under "remove deletes key from object"; the labeler found that `remove_kv` stops at the first match, so the comment's promise is already broken while the proof passes.
- Bend 2 tuning, from labels: a `case` pattern's literal is not a hardcoded-value candidate, nor are a zero-argument def's number or the samples of a `Bool` predicate that laws check; the value-kind follow-up offers a bound that only needs to be large enough (a recursion's fuel, an array's depth) and an arbitrary mixing constant, which took 61 wrong considers for 13 right ones; the function questions state Bend's shapes (one def per state machine, helper defs for computed matches, proofs that follow their definition) and flattening proposes nested patterns and a `case _:` fallback instead of guard clauses and early returns (function-simplification findings from 24 right and 16 wrong to 19 and 3).
- Parsing stops after 10 seconds and the file is skipped with that reason: Bend 2's grammar took over ten minutes on a 1 MB Bend 1 test.
- A request the provider refuses as beyond the model's context no longer fails its file: the units it asked that no other request answered need context, as a unit the budget does not send, and a refused follow-up leaves its unit with the answers it has. A Bend 2 proof of SHA-256 and a 328-member outline were refused, which made both runs incomplete. Outlines are also estimated at no more than 2.0 bytes per token, since their member lists tokenize at about 2.2 (against about 3 for code).

## [0.21.0] - 2026-09-26

Measured on 103 pinned projects (24 new open-source ones of kinds not tried before, among them intentionally vulnerable Rails, Node, GraphQL, C# and Java apps, a Deno framework, a WordPress plugin, a cookiecutter template and projects in Kotlin, Swift, Elixir and C, and 8 more of the maintainer's own), with findings labeled by hand: on the 72 labeled projects JevGate was tuned on, 76% of reviews were right against 69% with 0.20.0 (142 wrong reviews against 192), and 73% of considers against 64% (239 wrong considers against 355); on 11 held-out projects, 63% of reviews against 57%, and 56% of considers against 54%. On 14 projects added after that tuning (9 of the maintainer's own, Online Boutique, a browser extension, a React Native template, a Solidity and a dbt project), 64% of reviews and 65% of considers were right, against 53% and 47%. Undecided units went from 2.2% to 0.9% of judged units on those 103 projects.
Expand Down
11 changes: 11 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ tree-sitter-c-sharp = "=0.23.5"
tree-sitter-ruby = "=0.23.1"
tree-sitter-php = "=0.24.2"
tree-sitter-java = "=0.23.5"
tree-sitter-bend2 = "=0.1.2"
ureq = { version = "=3.4.2", default-features = false, features = ["rustls", "json"] }

# The JSON Schema of jevgate.toml is generated by a test from the configuration types.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Consider (2):
| Security | Injection, sensitive data, unsafe settings, SQL access control, GitHub workflows; each finding names a CWE | `--rule security` |
| Documentation | Agent instruction files, large and stale docs, duplicated sections, code comments | `--rule documentation` |

It reads Rust, Python, JavaScript, TypeScript, Go, C#, Ruby, PHP and Java, the scripts of Astro, Vue and Svelte files and the inline scripts of server templates (ERB, EJS, JSP, Handlebars, Jinja and others), SQL for PostgreSQL and Supabase, GitHub Actions workflows, and Markdown, MDX, reStructuredText and AsciiDoc, and knows the routes, handlers and settings of frameworks from Express, Next.js and SvelteKit to Django, Laravel, ASP.NET Core and Spring MVC. [What it finds](https://tech-byte-frontier.github.io/jevgate/what-it-finds.html) and [supported languages and frameworks](https://tech-byte-frontier.github.io/jevgate/languages.html) have the details; `jevgate rules` prints every rule with the question it asks.
It reads Rust, Python, JavaScript, TypeScript, Go, C#, Ruby, PHP, Java and Bend 2, the scripts of Astro, Vue and Svelte files and the inline scripts of server templates (ERB, EJS, JSP, Handlebars, Jinja and others), SQL for PostgreSQL and Supabase, GitHub Actions workflows, and Markdown, MDX, reStructuredText and AsciiDoc, and knows the routes, handlers and settings of frameworks from Express, Next.js and SvelteKit to Django, Laravel, ASP.NET Core and Spring MVC. [What it finds](https://tech-byte-frontier.github.io/jevgate/what-it-finds.html) and [supported languages and frameworks](https://tech-byte-frontier.github.io/jevgate/languages.html) have the details; `jevgate rules` prints every rule with the question it asks.

## Install

Expand Down
6 changes: 6 additions & 0 deletions jevgate.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@
"tests/redundancy",
"redundancy",
"test_redundancy",
"tests/laws",
"laws",
"documentation/agent-context",
"agent-context",
"agent_context",
Expand Down Expand Up @@ -127,6 +129,8 @@
"tests/redundancy",
"redundancy",
"test_redundancy",
"tests/laws",
"laws",
"documentation/agent-context",
"agent-context",
"agent_context",
Expand Down Expand Up @@ -217,6 +221,8 @@
"tests/redundancy",
"redundancy",
"test_redundancy",
"tests/laws",
"laws",
"documentation/agent-context",
"agent-context",
"agent_context",
Expand Down
2 changes: 1 addition & 1 deletion site/generate.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
COMMANDS = ["auth", "check", "baseline", "rules", "init", "completions", "man", "serve", "mcp"]
GROUPS = {
"maintainability": "On by default.",
"tests": "On by default; judged with `--include-tests` or `include_tests = true`.",
"tests": "On by default. Test value and test redundancy are judged with `--include-tests` or `include_tests = true`; the laws of Bend 2 code are judged without it.",
"security": "Opt-in: `--rule security`, or a level in `[rules]`.",
"documentation": "Opt-in: `--rule documentation`, or a level in `[rules]`.",
}
Expand Down
2 changes: 2 additions & 0 deletions site/src/languages.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
| Ruby | `.rb` | ✅ | ✅ RSpec, Minitest, Rails `test "…" do` | ✅ no Ruby framework handlers yet | ✅ comments |
| PHP | `.php` `.phtml` | ✅ | ✅ PHPUnit `…TestCase` classes, Pest `test`/`it` | ✅ | ✅ comments |
| Java | `.java` | ✅ | ✅ JUnit 4 and 5, TestNG: `@Test`, `@ParameterizedTest`, `@Nested`, JUnit 3 `TestCase` | ✅ | ✅ comments |
| Bend 2 ([bendlang/bend](https://github.com/bendlang/bend) 2.0.x) | `.bend` | ✅ | ✅ programs ending in the `#\|` lines their run must print, or defining `main` on a test path; laws (`tests/laws`) | ✅ defs that perform effects or build text | ✅ comments |
| Astro, Vue, Svelte | `.astro` `.vue` `.svelte` | ✅ scripts only | ➖ | ✅ scripts only | ✅ script comments |
| Server templates: ERB, EJS, JSP, Handlebars, Mustache, Nunjucks, Twig, Jinja, Go | `.erb` `.ejs` `.jsp` `.hbs` `.mustache` `.njk` `.twig` `.jinja` `.j2` `.tmpl` `.gohtml`, and `.html` under `templates/`, `views/`, `layouts/`, `partials/` or `includes/` | ✅ inline scripts only | ➖ | ✅ inline scripts, as the page's code in the visitor's browser; and the code that reads the request, a cookie, the session or the signed-in user: tags that write it unescaped (`<%= raw … %>`, `.html_safe`, `<%== … %>`, `<%- … %>`, `{{{ … }}}`, `\|safe`, `\|raw`) and a JSP page's scriptlets | ✅ script comments |
| SQL (PostgreSQL, Supabase) | `.sql` | ➖ | ➖ | ✅ access control | ➖ |
Expand Down Expand Up @@ -48,6 +49,7 @@
| Bundlers and compilers | Minified and compiled output (a source map reference, very long lines) is skipped as generated |
| Copied libraries | A library copied into the repository (a versioned file name such as `jquery-3.6.0.js`, the readable build beside a `.min.js`, a license banner naming a version, or a script under `assets`, `static` or `vendor` that opens with a whole license and copyright) is skipped as vendored, whatever its size |
| Project templates (cookiecutter, copier) | Files under a directory named with a `{{ … }}` placeholder are parsed without their Jinja tags, so the generated project's code is judged instead of skipped for syntax errors |
| Bend 2 | Defs, types and laws are units, named with their dots (`List.map`), and a call through an import alias (`Sort.sort` with `import ./main.bend as Sort`) reaches the def it names. A law is a claim when it states an equality, asks for a witness or applies a def that computes a type, and the def of its name is its proof; proofs (including every def of a `PROOF.bend`, a `*_proof.bend` or a file under `proofs/`) are not asked to be split, proofs and type-level defs are not asked about hardcoded values, and only defs that perform effects (`IO`) or join text with `++` are asked the security questions, since the rest are pure. `LAWS.bend`, `PROOF.bend` and files of fewer than 300 member lines are not asked to be split, a split of a file is at most a consider and a note in a file laid out in titled sections, a benchmark's values are not asked about, a program on a test path that defines `main` is a test, a zero-argument def returning a number names it, a `base.bend` copied from Bend's Base library is vendored, and Jev is told Bend 2's notation beside each file. Bend 1 files, a different language with the same `.bend` extension, are skipped with that reason |
| Migrations | Directories named `migrations`, Rails' `db/migrate` and timestamped scripts under `db/`, and Alembic's `alembic/versions` are skipped as migrations; SQL migrations are still read for access control |

Other files, such as Kotlin, are listed as skipped with the reason and never fail the gate.
2 changes: 1 addition & 1 deletion site/src/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Accept it with `jevgate baseline`, and record why with `jevgate baseline mark wr

## A file is skipped

Skipped files are listed with the reason: generated, vendored or minified code, migrations, an unsupported language, or a path outside the upload patterns. `generated`, `tests` and the upload patterns in `jevgate.toml` change what is selected. A file larger than `max_file_bytes` is not skipped but reported as `needs-context`, never truncated.
Skipped files are listed with the reason: generated, vendored or minified code, migrations, an unsupported language, syntax errors, a parser that did not finish within 10 seconds, Bend 1 code (JevGate reads Bend 2), or a path outside the upload patterns. `generated`, `tests` and the upload patterns in `jevgate.toml` change what is selected. A file larger than `max_file_bytes` is not skipped but reported as `needs-context`, never truncated, and so is a unit whose request the provider refuses as beyond the model's context.

## No colors, or escape codes in a log

Expand Down
3 changes: 2 additions & 1 deletion site/src/what-it-finds.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,13 @@
| Shared logic | `createInvoice` and `createReceipt` perform the same steps; one shared implementation would serve both. |
| Hardcoded values | Module constants fix a value that differs between deployments; `apply_discount` special-cases one specific customer. |

**Tests** (with `--include-tests`; file organization judges test files without it)
**Tests** (with `--include-tests`; file organization judges test files without it, and the laws of Bend 2 code are judged where they are)

| Rule | Example finding |
|---|---|
| Test value | `test_total` computes its expected value with the logic it tests. |
| Test redundancy | Three tests of `parse_date` check the same behavior; one parameterized test could hold them. |
| Laws (Bend 2) | The comment above law `body_after_blank` promises more than the law states: it says the text after a blank line is the body, and the law checks only texts that open with the blank line, so an `http_body` that returns a real response's headers could pass every proof. |

**Security** (opt-in with `--rule security`; each finding names a CWE)

Expand Down
Loading
Loading