fix(engine): relational compare over string-bound operands must fail safe, not lexically - #37
Open
konih wants to merge 2 commits into
Open
fix(engine): relational compare over string-bound operands must fail safe, not lexically#37konih wants to merge 2 commits into
konih wants to merge 2 commits into
Conversation
…l safe, not lexically CEL defines < <= > >= over strings as a LEXICAL compare, so an ordering leaf silently answered a boolean whenever its operands bound as text -- and both answers are wrong. Reproduced end to end through the production entry point (change.Diff -> evaldecode.DecodeCanonical -> aggregate.Cover) on a routine adopter shape: a quoted `partitions: "12"` -> `"6"` is a !!str the differ tag-discriminates and the decoder keeps a Go string by design, so the D-016 partitions-must-not-shrink rule evaluated the lexical "6" >= "12" = true, proved non-destructive, and returned APPROVE with zero findings. The mirror case is equally wrong: a legitimate grow "6" -> "12" evaluated false and BLOCKed. evalLeaf now plants a watcher on every operand of every relational operator; an operand that ACTUALLY evaluates to text makes the leaf an evaluation error -> predicate.error -> REVIEW. Value-based, not syntax-based: whether `new >= old` is sound depends on the adopter data, so lint cannot decide it. Watching (rather than reading post-eval EvalState, which keeps one value per node id) makes every comprehension iteration visible; a short-circuited compare never runs its watcher, so a policy cannot flip to REVIEW for a compare that did not happen. toCEL also stops demoting an unrepresentable numeric literal to its string form (9e399 > 1e400 was lexically true, numerically false); it binds a CEL error value instead. The over-int64-but-float64-representable lossy case is unchanged and still live. Behaviour change: a policy that previously evaluated (wrongly) now errors. Nothing in examples/, the comparison corpus, either dogfood pack or the test corpus orders text, so no golden or gate output changed. Ordering quoted numerics means coercing first -- int(new) >= int(old). ADR-0013 Amendment 1, D-129.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes a P1 decision-path fail-open: CEL's relational operators are defined over strings and compare them lexically, so an ordering leaf silently answered a boolean whenever its operands bound as text. Base
e54a243(lane started from163e91d, rebased).Reproduction (done first, both levels, before any fix)
End-to-end through the production chain (
change.Diff→evaldecode.DecodeCanonical→aggregate.Cover), on a routine adopter shape — a quoted numeric. New fixtureinternal/evaldecode/testdata/shrink-diff-quoted/{base,head}ispartitions: "12"→"6"; that is a!!str, so the differ renders it JSON-quoted and the decoder keeps it the Go string"12"by design (a quoted"12"is not the number 12):The D-016
partitions-must-not-shrinkrule (when: new >= old) evaluated the lexical"6" >= "12"= true, provednon-destructive, fired nothing → BLOCK → APPROVE flip, zero findings.The mirror case is equally wrong and is now pinned as its own test: a legitimate grow
"6"→"12"evaluated"12" >= "6"= false, so the rule fired and BLOCKed a change the policy permits:Unit level (all four operators, both argument orders, all returned a boolean with
err == nil):new >= old→ true,new <= old→ false,old >= new→ false, … plusentry >= oldEntry,mr.author < path, and the overflow arm9e399 > 1e400→ true (numerically false), fromtoCEL'sx.String()fallback.Design choice — value-based guard at the evaluator, strict, no exemption
evalLeafbuilds its program with acel.CustomDecoratorthat plants a watcher on every operand of every relational operator (<,<=,>,>=). If an operand actually evaluates to a string, the leaf returns an error →predicate.error→ REVIEW.new >= oldis sound is a property of the adopter's data, not of the policy text: correct over numbers, unsound over strings, and identical source either way. I checkedcheckLeafScope,checkPredicateScopeandcheckFactsShape— none could refuse this without also refusing the D-016 rule the whole corpus depends on. Lint sees the rule; it never sees the change it will judge. So no lint rule was added, deliberately.EvalState. My first implementation usedcel.OptTrackStateand had a real hole: state keeps one value per AST node id, sochanges.all(c, c.new > c.old)over a text element followed by a numeric one came back a cleantruewith the text compare overwritten and invisible. Watching sees every iteration.TestComprehensionTextCompareCaughtEveryIterationpins it in both element orders.kind == "add" && new >= oldstays a clean false).string(...)as "declared intent". Rejected: it re-admits the identical defect —new >= "12"withnewbound to!!str "6"is still lexically true, still APPROVE. A quoted numeric in a policy leaf is the same authoring slip as a quoted numeric in adopter YAML. Strict is also the smaller diff with no second code path.evalRuleas well (the exported walking-skeletonAggregatepath, test-only today), whose env declaresold/newasStringTypeand binds the raw canonical strings — every bare relational there was lexical by construction, with only convention mandatingint(). One unguarded evaluator is how this class comes back; same drift argument that pulled the decoder intointernal/evaldecode(D-055c).EvalScalar.{{ }}message interpolation renders text and decides nothing; guarding it would widen blast radius for no safety gain.toCELsecond arm. A numeric literal representable as neither int64 nor float64 now binds a CEL error value instead of its string form. The over-int64-but-float64-representable lossy case is unchanged and still live — that half of ADR-0013 residual build(deps): bump gitleaks/gitleaks-action from 2 to 3 #1 stays open, and the docs now say exactly that.Legitimate string relational compares — the search, and the answer
Searched
examples/(policies, packs, comparison corpus, lint fixtures, repo corpus), the adopter packs,cmd/assent/testdata,internal/adoptertest/testdata, the ADRs and every Go test. Every relational leaf in the repo is numeric:new >= old/new <= facts.quota.max_partitions.value(partitions),new >= facts.band.memory_mb.value.min,new >= entry.min_replicas(replicas/memory),size(...) > 0, and the already-coercedint(new) <= facts.quota.max.value. Nothing orders text — no ISO-date, semver or name ordering anywhere. (Worth noting the julieops corpus does carry quoted numerics —num.partitions: "1"— which is the real-world evidence that the reachable case is routine, not exotic.)So the strict rule breaks nothing that exists, and deliberate ordering stays expressible by coercing first:
int(new) >= int(old)(already the repo idiom),double(...), ortimestamp(a) < timestamp(b)for ISO dates. What is genuinely foreclosed — stated plainly rather than hidden — is ordering raw text in tier-1assert; that graduates to the Rego escape hatch, consistent with ADR-0013's "don't grow a programming language in YAML". Equality,inand the string functions are untouched, andTestLegitimateComparesStillEvaluatepins twelve of them still evaluating to the right booleans.Changed outputs — the null result, stated affirmatively
No golden, fixture, corpus or gate output changed. D-016 replay, the conformance suite, both comparison suites + the E6 seed, both dogfood packs (
service-catalog,infra-vars), the coverage gate and the determinism gate are all green unchanged — expected, given the search above.Exactly one existing test changed, and it is a finding rather than a regeneration:
internal/evaldecode'sTestStringOldNewFailsOpenasserted APPROVE to document the lexical fail-open as the decoder's mutation proof. That mutation now degrades to REVIEW instead of flipping to APPROVE, so the old assertion was blessing behaviour that no longer exists. RenamedTestUndecodedStringOldNewFailsSafeand rewritten to assert the discriminating pair rather than the weaker "not APPROVE" (which would pass with the decoder deleted):json.Number)partition-count-shrunkpredicate.errorDelete the decoder and the first arm reds; delete the guard and the second arm goes back to APPROVE. Neither layer can rot silently.
Mutation evidence
Every mutation applied, run, and reverted with
git diffconfirming the edit landed:textOrderGuard.recordneutered → 6 tests red across two packages (TestQuotedNumericShrinkMustNotApproveThroughCover,TestQuotedNumericGrowMustNotBlockLexically,TestRelationalOverStringOperandsErrors,TestComprehensionTextCompareCaughtEveryIteration,TestUndecodedStringOldNewFailsSafe,TestQuotedNumericShrinkFailsSafeEndToEnd).cel.CustomDecorator(guard.decorate)removed from theProgramcall → the same 6 red. (This one documents the fragile plan-time injection for a future cel-go bump.)toCEL'stypes.NewErrreverted tox.String()→TestToCELNeverYieldsAStringForANumericLiteralandTestUnrepresentableNumericFailsSafered. Note the layering visible here: with the string form back, the relational arms were still caught by the guard and onlynew == "9e399"slipped — which is precisely why both arms exist.RED-before-GREEN is in the history of this lane: every new test was written first and observed failing with the exact wrong values quoted above;
-vused throughout so a non-matching-runcannot masquerade as a pass.cel-go interaction check (v0.30.0):
textOrderWatchwraps operands as a plainInterpretable, unlike cel-go's owndecObserveEvalwhich preservesInterpretableAttribute/InterpretableConst. Safe because a relational operand's only consumer is the binary call itself — and proven, not merely argued:TestEvalRuleEnforcesCostBudget'scostBombWhenputs the expensive comprehension as the left operand of> 0, so it is exactly a wrapped node, and the cost budget still fires. A comment now pins that shape so a future rewrite doesn't silently drop the proof.Docs truth (F3) — and a second stale claim found
internal/evaldecode/evaldecode.go: the package doc claimed a value "must NEVER silently become a lexical string … never a permissive string" while a!!strdid exactly that. It now says what is true: decoding cannot close this, because a genuinely-textual value legitimately stays text; the engine seam closes it, and the two layers are complementary and both load-bearing. The LIMITATION no longer says "a lossy compare or its string form" — the string-form half is closed; the lossy half is named as still live.internal/core/aggregate's package doc still described the pre-evaldecodeworld — "They are bound to CEL as raw string values" and "numeric rules MUST use int()/double() in thewhenexpression". Both false since the decoder landed (the corpus is full of barenew >= old). Rewritten to the current truth plus the new guard.Records
main, leaving 126–128 for the concurrent session's uncommitted rows). It states the behaviour change plainly: a policy that previously evaluated (wrongly) now errors → REVIEW.assertand that named this exact residual ("numeric type coercion YAML/HCL→CEL — highest risk"). Follows the ADR-0007 Amendment 2 / ADR-0011 Amendment 3 precedent.docs/planning/predicate-scope.mdgains the authoring rule (that file is the frozen authoring contract).openspec/change entry — judged and stated:openspec/changes/contains only itsREADME.md, so this repo has never used change proposals in practice, and a fail-safe correction of an existing frozen semantic (no new contract surface, no schema change) is proportionate to failing test first → D-row → ADR amendment.schemas/is byte-unchanged (the compare exit gate'sgit diff schemas/guard proves it).docs/planning/open-questions.mdate54a243— checked at lane start and again at PR time; the concurrent lane has not landed it. I did not invent a row that would collide. If it lands before this merges, mark it resolved → D-129; if it does not, D-129 carries the resolution on its own.Gates
task checktask determinismtask dogfood-examplesbash hack/compare/exitgate_test.shgit diff schemas/empty)CI=true go test ./...go vet -tags e2e ./...TestExecDigestPintask checkwas run again after the finaltask changelog-writecommit, and is green at the pushed tip.Risks
cel.CustomDecorator+interpreter.Interpretable). A cel-go major bump should re-run mutation 2 and the cost-budget test. Mitigated by pinning the reasoning in comments at both sites.Aggregate/evalRulehas no non-test caller today; guarding it is defensive rather than load-bearing, and it is the one part of the diff beyond the reported finding.Scope respected:
internal/provider/builtin/{repo_file,resource_owner}.go,.github/workflows/**,Taskfile.yml,cliff.tomlandinternal/forge/**untouched.