Matcher assertions: assert/expect over equals/contains/not/isOk - #240
Closed
assapir wants to merge 27 commits into
Closed
Matcher assertions: assert/expect over equals/contains/not/isOk#240assapir wants to merge 27 commits into
assapir wants to merge 27 commits into
Conversation
…ed entry A top-level `describe(...)` call parses into `Program.test_blocks` rather than `items`, so every command but `quilon test` compiles the file without its tests. `quilon test` synthesizes the entry point that runs each block in order and ends with the reporter's summary. The registry (`quilon-rt::test_registry`) holds what a run needs and Quilon has no storage for: nesting depth, the case in progress, and the pass/fail totals. It renders nothing — that is the reporter seam. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`quilon test [path]` finds every suite under a path — a `.qn` file with top-level `describe` blocks — and JITs each on its own thread, exiting non-zero if any case failed. `build`/`compile`/`run` skip a file that is nothing but tests. `core.test` gains the harness: `describe`/`it` over `() -> $` closures, and an `expect` overload per scalar type returning a matcher that remembers where the `expect` was written, so a failure blames your call. Matchers RENDER AND CONTINUE — the run reports every failing case, not just the first — reusing the diagnostic frame that `failAt` already drew, now shared as `renderFrame`. Also fixes a `-g` panic: an opaque DWARF pointee had an empty name, which LLVM rejects, so any program holding a function value crashed under `--debug`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rust gates for both halves: the parse (a top-level `describe` CALL is a test block, a `describe =` definition is not), the strip (a release build of a mixed file emits neither `describe` nor an `expect` overload nor the reporter; a tests-only file is passed over in silence), and the run (a passing suite exits 0, a failing case exits non-zero without stopping the run and reports at its own `expect`, a directory runs each suite with its totals kept separate). Docs: `docs/corelib/test.md` gains the harness, the matcher table, the reporter seam, and the stripping rule; LANGUAGE.md and CLAUDE.md gain `quilon test`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The maintainer is redesigning `expect`, so the matcher records, their trailing-`Site` plumbing, `reportFailure`/`renderFrame`, and the three intrinsics that only served them are gone. A case now asserts with `assert`/`assertEq`/…, which are fail-fast: the first failure reports and exits 101, so the reporter says what it can know — every case up to that point, and a total only on a clean run. No "N passed, M failed" tally is claimed. The registry is down to four counters (`__test_suite_enter`/`_leave`, `__test_case_passed`, `__test_passed`), and the reporter seam to three functions. Suites now run one process each, so a fail-fast exit ends its own suite rather than the whole run. Review findings applied: an unparseable suite is run (and reported) instead of silently vanishing and passing; discovery no longer follows symlinks, so a link back up the tree cannot recurse forever; a suite keeps its fixtures without losing the silent skip; a missing reporter is reported at the `describe` rather than at a synthesized span; the tests-only check reuses the front end's parse instead of doing its own; `quilon test` no longer offers an argument passthrough the synthesized parameterless `^` could never read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With the matchers gone, a release build of a mixed file emits NOTHING of the harness — the stripping test now asserts that over the whole define list rather than a handful of names. And a `-g` gate for the opaque DWARF pointee: `examples/higher_order.qn` under debug info is a program holding a function value, which is what used to panic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A mistyped path in a CI invocation used to discover no suites and exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`examples/tests_alongside_code.qn` keeps its `describe` blocks next to the `>>` exports they check, and one case prints a line nothing else in the repository prints — so a build that shows it would be a build that compiled test code. `examples/uses_tested_module.qn` imports those exports and prints a marker of its own, which is what tells "the blocks were erased" apart from "nothing ran". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The examples gate only checks exit codes, so the stripping was asserted from the inside (no `describe` in the emitted IR) and never observed. Three tests now watch what a run prints: - a `describe` block beside an `^`, inline: `run` and a native build print the program's marker and never the block's line. - the shipped module: `run` of it says nothing, and `run`/`build` of the program that imports its exports print the marker without the block's line. - the same module under `quilon test`: that line IS on stdout and its case is reported — one file, two commands, opposite outcomes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ld's The cost side was documented; the consequence was not. `check`/`compile`/`build` strip the block before the checker sees it, so a broken test compiles clean and reports success — only `quilon test` catches it. Says so, says to run `quilon test` in CI, and points "tests beside the code" at the module form that `quilon test` will actually run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The compiler provides both entry points and the matcher vocabulary, so one matcher name works over every type without generics. `assert` reports at the call site and exits 101; `expect` reports, marks the running case failed, and returns — and reads that mark first, so a failure skips the rest of its case while the suite carries on. `expect` outside a `describe` block is refused at compile time, since there is no reporter to record into. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review findings. An imported module's test blocks are dropped at link time whatever the mode, so the pair test was never evidence of stripping: it is evidence about the import boundary, and now says so in its name. The same-file proof is the sibling test, which is where the claim belongs. Also: reuse `common::tool_available` instead of a fourth private copy, collapse the thrice-repeated marker assertions into one helper, name the constant for what it marks, pin the suite's case count so a dropped case fails, and follow the `mathlib`/`use_module` convention — `use_tested_module.qn`. The docs sentence gains `run` (it strips too) and drops the overstatement that `quilon test` refuses any file with an `^`: it refuses one that also has blocks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every matcher over the built-ins and over a user record and sum, negation, the Result matchers, and the diagnostics that refuse a matcher a type cannot answer. On the harness side: a failed expect skips the rest of its case and nothing more, the run tallies both ways, expect outside a describe block is refused, and assert stays fatal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs/corelib/test.md is rewritten around the two entry points and the five matchers; the fail-fast limitation is gone, replaced by what a failed expect actually does to its case. LANGUAGE.md, the roadmap item, CLAUDE.md and the example suite follow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A file with top-level `describe` blocks and its own `^` was rejected by `quilon test`, which left tests written next to an executable's entry point dead: erased from every build and unrunnable. They now work like Rust's `#[cfg(test)]` neighbours — every command but `quilon test` erases the blocks and the file's `^` is its entry point as usual, while `quilon test` compiles the blocks and runs them under the entry point it synthesizes, dropping the file's own so the two never collide on one symbol. The shipped example keeps its exports, its `^`, and its tests in one file and prints a distinct line from each half, so which line comes out says which halves were compiled.
Only `quilon test` compiles a `describe` block, so a type error inside one passed every gate the workflow had. Each platform job now runs the built binary over `examples/` after its `cargo test`, where a failing case or a suite that does not compile fails the run.
Two defects the correctness pass found. An expect directly in a describe
body had no case to mark: it printed a failure the summary never counted,
and poisoned the next case, whose assertions the stale mark then skipped.
It now needs an enclosing it, and says so. And a not-yet-concrete sum
payload is represented as a Num while the checker treats it as compatible
with anything, so equals("x") on one type-checked and handed codegen an f64
and a Text; the two sides must now share one representation.
Alongside, the simplification pass: the Text field split, the sum-tag
comparison and the i64-to-Bool narrowing each live in one place again, an
array element's slot takes its value representation, and the comments that
named the removed assert* family say what they mean now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The base branch's new example and harness tests move to the matcher form — an it case records with expect, an ^ asserts — and the two summary expectations become the real tally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
assapir
force-pushed
the
feat/test-harness
branch
from
August 27, 2026 07:58
346e5f1 to
4bee68d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PARKED — do not merge. Stacked on #233 (
feat/test-harness), which is its base.Implements the locked assertion redesign: an assertion takes the value under test first and a
matcher second, and the old
assert(x)/assertEq(a, b)family is gone.Shape
assert/expectand the five matchers (equals,contains,not,isOk,isNotOk) arecompiler-provided, like
print: no import, and a matcher is a form rather than a value —which is what lets one matcher name work over every type while the language has no generics.
src/typechecker/checker/assertions.rs) settles the matcher's shape and thetypes it can read:
equalsneeds an==member (so any user record or sum works exactly asfar as its own member does),
containsneeds aTextor an array,isOk/isNotOkneed asum carrying that variant,
notrecurses. A matcher applied to a type it cannot read is acompile error naming what is missing.
src/codegen/generator/assertions.rs) evaluates the subject once and lowersthe matcher to the condition it tests plus the description of what it wanted — rendered
only on the failing path, through the same
`render pathprintuses, with aTextquoted so a trailing space shows.
__assert_failed(frame + exit 101) and__expect_failed(frame + mark the case failed + return), both reusing
report.rs's existing frame renderer,which
fail_atnow splits intoreport_at+ exit.expectreads the case's failed mark first and evaluates nothingwhen it is set, so the first failure in a case skips the rest of that case — subjects
included — while the suite carries on. That is the isolation mechanism; no fibers involved.
expectneeds a case: legal only lexically inside anitinside adescribe. Outside adescribethere is no reporter (the blocks are stripped fromrun/compile/build);inside one but outside an
itthere is no case to mark, so the failure would print and neverbe counted. Both are compile errors pointing at
assert.What #233 could not do, and now can
A failure no longer ends the run, so the reporter prints a real tally.
reportCasegained afailed :: Bool, the registry gained a per-case mark and a failed count, andreportSummaryexits non-zero when any case failed:
Migration
Wide and mechanical. Every
examples/*.qn(all self-asserting), the.qnsnippets embedded inten Rust test files, both bench corpora, and
corelib/test.qn— which keeps the harness, thereporter and
failAt, and losesassert/assertEq/assertNotEq/assertOk/assertNotOk/AssertOpts.assertEq(a, b)→assert(a, equals(b)),assertNotEq(a, b)→assert(a, not(equals(b))),assertOk(r)→assert(r, isOk()),assert(a == b)→assert(a, equals(b)). A custom failure message has no replacement: the matcher's own reportnames what was expected and what was found.
Two gates moved with it: the examples self-assert check now looks for
assert(rather than a<< core.testimport, and the every-intrinsic link gate exempts__expect_failed(no^program can reach it —
expectonly exists in a suite, and suites run under the JIT only),pointing at where it is covered.
Ships with
tests/assert_test.rs— every matcher over the built-ins and over a user record and a usersum (its own
==decides, its own`renders), negation and double negation, theResultmatchers,Textquoting, subject-evaluated-once, the full failure frame, thelocation through a helper and through an imported module, native-AOT parity, and each
diagnostic that refuses a matcher a type cannot answer.
tests/test_harness_test.rs— a failedexpectskips the rest of its case and nothing more,the run tallies both ways and exits non-zero,
expectoutside adescribeand outside anitare both refused,assertin a case is still fatal.examples/assert_demo.qnrewritten around the vocabulary (including a user record with==and
`);examples/test_suite.qnnow showsexpect.docs/corelib/test.mdrewritten;docs/LANGUAGE.md,docs/ROADMAP.md(thequilon testitem flips 🔨 → ✅),
CLAUDE.mdandCHANGELOG.mdfollow.Review
Read-only correctness and simplification passes ran on the branch; both sets of findings are
addressed in
33b3adb. The correctness pass found two real defects — anexpectoutside anyitprinted a failure nothing tallied (suite exited 0) and poisoned the next case, and ageneric sum payload paired with a
Textpanicked inkwell instead of erroring. Both are fixedwith regression tests. It reported the block structure, dominance,
i1/i64/f64handling,tag convention, evaluation order, unsafe pointer handling and intrinsic wiring clean.
The simplification pass found five real duplications; the Text field split, the sum-tag
comparison and the
i64→Boolnarrowing now each live in one place. Two of its proposals weredeliberately not taken: splitting
generate_binary_operatorto share its value-level tail(the caveat is a worse error message on an unreachable path), and folding
green/redinto apainthelper (a module's non-exported names are not linked into a program, so it would need athird exported name).
Gate
cargo fmt --all -- --check,cargo clippy --workspace --all-targets --all-features -D warnings, andRUSTFLAGS=-D warnings cargo test --workspace— all green (46 test binaries).🤖 Generated with Claude Code