Skip to content

Matcher assertions: assert/expect over equals/contains/not/isOk - #240

Closed
assapir wants to merge 27 commits into
feat/test-harnessfrom
feat/matcher-assertions
Closed

Matcher assertions: assert/expect over equals/contains/not/isOk#240
assapir wants to merge 27 commits into
feat/test-harnessfrom
feat/matcher-assertions

Conversation

@assapir

@assapir assapir commented Aug 27, 2026

Copy link
Copy Markdown
Owner

PARKED — do not merge. Stacked on #233 (feat/test-harness), which is its base.

Implements the locked assertion redesign: an assertion takes the value under test first and a
matcher second, and the old assert(x) / assertEq(a, b) family is gone.

assert(2 + 2, equals(4))              ~ fatal: report at the call site, exit 101
expect(body, contains("HTTP/1.1"))    ~ recorded: mark the case failed, carry on
expect(status, not(equals(500)))
expect(response, isOk())

Shape

assert/expect and the five matchers (equals, contains, not, isOk, isNotOk) are
compiler-provided, like print: no import, and a matcher is a form rather than a value —
which is what lets one matcher name work over every type while the language has no generics.

  • Checker (src/typechecker/checker/assertions.rs) settles the matcher's shape and the
    types it can read: equals needs an == member (so any user record or sum works exactly as
    far as its own member does), contains needs a Text or an array, isOk/isNotOk need a
    sum carrying that variant, not recurses. A matcher applied to a type it cannot read is a
    compile error naming what is missing.
  • Codegen (src/codegen/generator/assertions.rs) evaluates the subject once and lowers
    the matcher to the condition it tests plus the description of what it wanted — rendered
    only on the failing path, through the same ` render path print uses, with a Text
    quoted so a trailing space shows.
  • Runtime: two new intrinsics, __assert_failed (frame + exit 101) and __expect_failed
    (frame + mark the case failed + return), both reusing report.rs's existing frame renderer,
    which fail_at now splits into report_at + exit.
  • The skip semantics: expect reads the case's failed mark first and evaluates nothing
    when it is set, so the first failure in a case skips the rest of that case — subjects
    included — while the suite carries on. That is the isolation mechanism; no fibers involved.
  • expect needs a case: legal only lexically inside an it inside a describe. Outside a
    describe there is no reporter (the blocks are stripped from run/compile/build);
    inside one but outside an it there is no case to mark, so the failure would print and never
    be counted. Both are compile errors pointing at assert.

What #233 could not do, and now can

A failure no longer ends the run, so the reporter prints a real tally. reportCase gained a
failed :: Bool, the registry gained a per-case mark and a failed count, and reportSummary
exits non-zero when any case failed:

arithmetic
  ✓ holds
  ✗ does not hold
  ✓ runs after the failure

2 passed, 1 failed

Migration

Wide and mechanical. Every examples/*.qn (all self-asserting), the .qn snippets embedded in
ten Rust test files, both bench corpora, and corelib/test.qn — which keeps the harness, the
reporter and failAt, and loses assert/assertEq/assertNotEq/assertOk/assertNotOk/
AssertOpts. assertEq(a, b)assert(a, equals(b)), assertNotEq(a, b)
assert(a, not(equals(b))), assertOk(r)assert(r, isOk()), assert(a == b)
assert(a, equals(b)). A custom failure message has no replacement: the matcher's own report
names what was expected and what was found.

Two gates moved with it: the examples self-assert check now looks for assert( rather than a
<< core.test import, and the every-intrinsic link gate exempts __expect_failed (no ^
program can reach it — expect only exists in a suite, and suites run under the JIT only),
pointing at where it is covered.

Ships with

  • tests/assert_test.rs — every matcher over the built-ins and over a user record and a user
    sum (its own == decides, its own ` renders), negation and double negation, the
    Result matchers, Text quoting, subject-evaluated-once, the full failure frame, the
    location through a helper and through an imported module, native-AOT parity, and each
    diagnostic that refuses a matcher a type cannot answer.
  • tests/test_harness_test.rs — a failed expect skips the rest of its case and nothing more,
    the run tallies both ways and exits non-zero, expect outside a describe and outside an
    it are both refused, assert in a case is still fatal.
  • examples/assert_demo.qn rewritten around the vocabulary (including a user record with ==
    and `); examples/test_suite.qn now shows expect.
  • docs/corelib/test.md rewritten; docs/LANGUAGE.md, docs/ROADMAP.md (the quilon test
    item flips 🔨 → ✅), CLAUDE.md and CHANGELOG.md follow.

Review

Read-only correctness and simplification passes ran on the branch; both sets of findings are
addressed in 33b3adb. The correctness pass found two real defects — an expect outside any
it printed a failure nothing tallied (suite exited 0) and poisoned the next case, and a
generic sum payload paired with a Text panicked inkwell instead of erroring. Both are fixed
with regression tests. It reported the block structure, dominance, i1/i64/f64 handling,
tag convention, evaluation order, unsafe pointer handling and intrinsic wiring clean.

The simplification pass found five real duplications; the Text field split, the sum-tag
comparison and the i64Bool narrowing now each live in one place. Two of its proposals were
deliberately not taken: splitting generate_binary_operator to share its value-level tail
(the caveat is a worse error message on an unreachable path), and folding green/red into a
paint helper (a module's non-exported names are not linked into a program, so it would need a
third exported name).

Gate

cargo fmt --all -- --check, cargo clippy --workspace --all-targets --all-features -D warnings, and RUSTFLAGS=-D warnings cargo test --workspace — all green (46 test binaries).

🤖 Generated with Claude Code

assapir and others added 27 commits August 26, 2026 14:49
…ed entry

A top-level `describe(...)` call parses into `Program.test_blocks` rather than
`items`, so every command but `quilon test` compiles the file without its tests.
`quilon test` synthesizes the entry point that runs each block in order and ends
with the reporter's summary.

The registry (`quilon-rt::test_registry`) holds what a run needs and Quilon has
no storage for: nesting depth, the case in progress, and the pass/fail totals. It
renders nothing — that is the reporter seam.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`quilon test [path]` finds every suite under a path — a `.qn` file with top-level
`describe` blocks — and JITs each on its own thread, exiting non-zero if any case
failed. `build`/`compile`/`run` skip a file that is nothing but tests.

`core.test` gains the harness: `describe`/`it` over `() -> $` closures, and an
`expect` overload per scalar type returning a matcher that remembers where the
`expect` was written, so a failure blames your call. Matchers RENDER AND CONTINUE
— the run reports every failing case, not just the first — reusing the diagnostic
frame that `failAt` already drew, now shared as `renderFrame`.

Also fixes a `-g` panic: an opaque DWARF pointee had an empty name, which LLVM
rejects, so any program holding a function value crashed under `--debug`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rust gates for both halves: the parse (a top-level `describe` CALL is a test
block, a `describe =` definition is not), the strip (a release build of a mixed
file emits neither `describe` nor an `expect` overload nor the reporter; a
tests-only file is passed over in silence), and the run (a passing suite exits 0,
a failing case exits non-zero without stopping the run and reports at its own
`expect`, a directory runs each suite with its totals kept separate).

Docs: `docs/corelib/test.md` gains the harness, the matcher table, the reporter
seam, and the stripping rule; LANGUAGE.md and CLAUDE.md gain `quilon test`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The maintainer is redesigning `expect`, so the matcher records, their
trailing-`Site` plumbing, `reportFailure`/`renderFrame`, and the three intrinsics
that only served them are gone. A case now asserts with `assert`/`assertEq`/…,
which are fail-fast: the first failure reports and exits 101, so the reporter
says what it can know — every case up to that point, and a total only on a clean
run. No "N passed, M failed" tally is claimed.

The registry is down to four counters (`__test_suite_enter`/`_leave`,
`__test_case_passed`, `__test_passed`), and the reporter seam to three functions.

Suites now run one process each, so a fail-fast exit ends its own suite rather
than the whole run.

Review findings applied: an unparseable suite is run (and reported) instead of
silently vanishing and passing; discovery no longer follows symlinks, so a link
back up the tree cannot recurse forever; a suite keeps its fixtures without
losing the silent skip; a missing reporter is reported at the `describe` rather
than at a synthesized span; the tests-only check reuses the front end's parse
instead of doing its own; `quilon test` no longer offers an argument passthrough
the synthesized parameterless `^` could never read.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With the matchers gone, a release build of a mixed file emits NOTHING of the
harness — the stripping test now asserts that over the whole define list rather
than a handful of names.

And a `-g` gate for the opaque DWARF pointee: `examples/higher_order.qn` under
debug info is a program holding a function value, which is what used to panic.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A mistyped path in a CI invocation used to discover no suites and exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`examples/tests_alongside_code.qn` keeps its `describe` blocks next to the `>>`
exports they check, and one case prints a line nothing else in the repository
prints — so a build that shows it would be a build that compiled test code.
`examples/uses_tested_module.qn` imports those exports and prints a marker of its
own, which is what tells "the blocks were erased" apart from "nothing ran".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The examples gate only checks exit codes, so the stripping was asserted from the
inside (no `describe` in the emitted IR) and never observed. Three tests now watch
what a run prints:

- a `describe` block beside an `^`, inline: `run` and a native build print the
  program's marker and never the block's line.
- the shipped module: `run` of it says nothing, and `run`/`build` of the program
  that imports its exports print the marker without the block's line.
- the same module under `quilon test`: that line IS on stdout and its case is
  reported — one file, two commands, opposite outcomes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ld's

The cost side was documented; the consequence was not. `check`/`compile`/`build`
strip the block before the checker sees it, so a broken test compiles clean and
reports success — only `quilon test` catches it. Says so, says to run `quilon test`
in CI, and points "tests beside the code" at the module form that `quilon test`
will actually run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The compiler provides both entry points and the matcher vocabulary, so one
matcher name works over every type without generics. `assert` reports at the
call site and exits 101; `expect` reports, marks the running case failed, and
returns — and reads that mark first, so a failure skips the rest of its case
while the suite carries on. `expect` outside a `describe` block is refused at
compile time, since there is no reporter to record into.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review findings. An imported module's test blocks are dropped at link time
whatever the mode, so the pair test was never evidence of stripping: it is
evidence about the import boundary, and now says so in its name. The same-file
proof is the sibling test, which is where the claim belongs.

Also: reuse `common::tool_available` instead of a fourth private copy, collapse
the thrice-repeated marker assertions into one helper, name the constant for what
it marks, pin the suite's case count so a dropped case fails, and follow the
`mathlib`/`use_module` convention — `use_tested_module.qn`.

The docs sentence gains `run` (it strips too) and drops the overstatement that
`quilon test` refuses any file with an `^`: it refuses one that also has blocks.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every matcher over the built-ins and over a user record and sum, negation,
the Result matchers, and the diagnostics that refuse a matcher a type cannot
answer. On the harness side: a failed expect skips the rest of its case and
nothing more, the run tallies both ways, expect outside a describe block is
refused, and assert stays fatal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs/corelib/test.md is rewritten around the two entry points and the five
matchers; the fail-fast limitation is gone, replaced by what a failed expect
actually does to its case. LANGUAGE.md, the roadmap item, CLAUDE.md and the
example suite follow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A file with top-level `describe` blocks and its own `^` was rejected by
`quilon test`, which left tests written next to an executable's entry point
dead: erased from every build and unrunnable. They now work like Rust's
`#[cfg(test)]` neighbours — every command but `quilon test` erases the
blocks and the file's `^` is its entry point as usual, while `quilon test`
compiles the blocks and runs them under the entry point it synthesizes,
dropping the file's own so the two never collide on one symbol.

The shipped example keeps its exports, its `^`, and its tests in one file
and prints a distinct line from each half, so which line comes out says
which halves were compiled.
Only `quilon test` compiles a `describe` block, so a type error inside one
passed every gate the workflow had. Each platform job now runs the built
binary over `examples/` after its `cargo test`, where a failing case or a
suite that does not compile fails the run.
Two defects the correctness pass found. An expect directly in a describe
body had no case to mark: it printed a failure the summary never counted,
and poisoned the next case, whose assertions the stale mark then skipped.
It now needs an enclosing it, and says so. And a not-yet-concrete sum
payload is represented as a Num while the checker treats it as compatible
with anything, so equals("x") on one type-checked and handed codegen an f64
and a Text; the two sides must now share one representation.

Alongside, the simplification pass: the Text field split, the sum-tag
comparison and the i64-to-Bool narrowing each live in one place again, an
array element's slot takes its value representation, and the comments that
named the removed assert* family say what they mean now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The base branch's new example and harness tests move to the matcher form —
an it case records with expect, an ^ asserts — and the two summary
expectations become the real tally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@assapir
assapir force-pushed the feat/test-harness branch from 346e5f1 to 4bee68d Compare August 27, 2026 07:58
@assapir
assapir deleted the branch feat/test-harness August 27, 2026 08:09
@assapir assapir closed this Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant