Test oracle is the printed string (706 vs 354 numeric): it pins the hash function, not the mathematics

[petlenz]

The test suite's dominant oracle is the printed string: 706 EXPECT_PRINT/EXPECT_SAME_PRINT assertions vs 354 numeric EXPECT_NEAR/EXPECT_DOUBLE_EQ. ScalarExpressionTest.h alone has 281 print-based assertions.

Why this is a design problem, not a style preference:

1. Print order is hash order (expression::operator< → hash_value(), src/numsim_cas/core/expression.cpp), so every print assertion silently pins the current hash function. Any change to hashing — even one that fixes a bug — flips dozens of unrelated tests. Concretely: the hash discriminator that would have fixed Scalar and tensor symbols with the same name collide (name-only hash, per-domain ids): assert abort in debug, UB with NDEBUG #348 cleanly was reverted in PR Fix #348: discriminate symbols across domains in expression identity #442 because 7–43 print-pinned tests flipped, and Sums are not canonical: like-term matching cannot pair a term with its scaled or negated form (282/1728 associativity triples differ) #468 (like-term canonicalisation) is blocked on the same cost. The tests are protecting the hash function, not the mathematics.

2. A print assertion cannot distinguish "same value" from "same shape". x + 2*x vs 3*x print differently and are different canonical forms — fine to pin. But 2*x*tr(A)-(3*x*tr(A)) vs -x*tr(A) (t2s scalar_wrapper equality is hash-only over a lossy hash: (2x)·tr(A) − (3x)·tr(A) folds to 0 #451's fixed result) are the same value and different shapes; a print test forces a choice the library has not made ([epic] Canonical form + hash-consing: make expression identity structural, not conventional #379: no defined canonical form).

3. Negative controls are ad hoc. Every fix this week proved its test fails on unfixed code by hand (git show main:file > file). Nothing enforces that a new test can fail; the print-string tests that passed on both sides of Scalar simplifier cleanups: mul_pow scalar copy misses empty-mul collapse; symbol_mul uses pointer identity for x*x #371 show the risk.

4. Fuzz was blind to 39 of 69 node classes until Fuzz differentiation generators omit if_then_else, max/min, comparisons, eigen, isotropic functions and most math functions #450/Fix #450: fuzz the node classes the generators never emitted #481; the deterministic suite had the same blind spots (the if_then_else segfault Comparing two if_then_else nodes segfaults — ternary_op has no comparison operators #457 lived in code that 706 print tests never compared).

Proposal:

• Add a structural-equality oracle and prefer it: EXPECT_EXPR_EQ(a, b) using deep operator==, and EXPECT_EXPR_VALUE_EQ(a, b, points...) evaluating both at sample points. Migrate print assertions that are really "these simplify to the same thing" onto them; keep EXPECT_PRINT only in printer tests, where the string is the contract.
• Decouple print order from hash order for tests: a deterministic, human-meaningful order for printing (e.g. by symbol name then structure) would let the hash change freely. This is also what codegen wants — stable emitted C++ across library versions.
• Mutation check in CI for new tests: a lightweight script that reverts the non-test hunks of a PR and asserts at least one new test fails. Cheap, and it turns the hand-run negative control into policy.
• Track the ratio: add a badge/test that fails if print-string assertions grow in non-printer test files.

Signed-off-by: petlenz peterlenz89.pl@gmail.com