Skip to content

Generated rows have no per-case coverage bound: lockfile leaf pinning #49

Description

@lann

From polymorph-components/polymorph-webcrypto#302 (adversarial review of the M1 migration; mechanics confirmed against this repo at c90b2f0): [[generated]] lockfile entries record prefix + tags only, and check_coverage holds generated leaves to grammar + at-most-once — no lower bound, no membership. A generated row reporting zero leaves passes coverage; a same-count case substitution passes everything. For corpora that are overwhelmingly generated (webcrypto: ~99.7%), the lockfile pins the inventory at prefix granularity only, and downstream's "the inventory is the binding" is stronger than the mechanism. This also undercuts the unknown-status tolerance story: a diverted unknown-status row under a generated prefix is a warning + exit 0 (polymorph-components/polymorph-webcrypto#303).

The structural obstacle: lock is execution-free by design (static section only), and generated leaves exist only at enumeration time — that is why prefixes exist. So any per-leaf pin needs an enumeration input. Options, roughly in order of preference:

  1. Leaf enumeration in the lockfile. [[generated]] grows an optional cases = [...] (or per-leaf lines). Restores the incumbent-style named-diff review surface — a reviewer sees which cases appeared/vanished, which a digest cannot show. Cost: large lockfiles (webcrypto: ~18k lines — the incumbent lock was that size and reviewed fine) and lock needing enumeration: either lock --leaves <file> fed from ct-runner --enumerate, or lock gaining an opt-in execution mode (a deliberate break from execution-free — worth deciding explicitly).
  2. Count + leaf-set digest per prefix. Compact; catches drops, additions, and substitutions; loses the named diff (opaque hash churn on any vector change). Same enumeration-input problem.
  3. Aggregate-side cross-target leaf-set agreement. No format change, no enumeration input: all targets ran the same suite artifact, so their generated leaf sets must be identical — divergence is an error. Catches per-target drops/filters but not a uniform regression (same artifact everywhere). Cheap and worth doing regardless of 1/2 as defense in depth.
  4. Downstream-only census fixtures per prefix (no upstream change) — the fallback if 1/2 are declined.

Interaction with check_coverage: with leaves (or count+digest) present, generated coverage becomes the same set-equality exact cases get; absent, current behavior (compatibility: old lockfiles keep working — additive format evolution).

I'd propose 1 + 3: enumeration as the pinned review surface, cross-target agreement as the structural check that needs no lockfile at all.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions