Six normative questions were deferred during the review of PR #63 (efficacy protocol, CodeRabbit findings 1, 4, 5, 8, 9, 11 — see the review-disposition comment there). The sitrep (docs/sitrep-2026-09-01.adoc, debt item D1) batches them into this one issue per the 2026-08-27 batching ruling.
How to rule: six one-line replies resolve the whole issue, e.g. a: 1, b: 1, c: 1, d: 1, e: 1, f: authorized. Anything not option 1 with a note is equally fine. After rulings, one PR amends vexometer/docs/EFFICACY-PROTOCOL.adoc to v2.1 with all examples updated — no normative text changes before then.
Live consequence while open: the D5 efficacy-report tool (in progress) will refuse to compute a zero-baseline gap_closed with an explicit awaiting ruling D1a error rather than silently picking a semantic.
All line references are to vexometer/docs/EFFICACY-PROTOCOL.adoc at bf2ec6d.
(a) G_m when the baseline is zero
Current wording (lines 40–47):
Gap closed (G_m): The fraction of the available irritation gap removed. Since the floor is 0, the available gap is the baseline itself:
G_m = (B_m - A_m) / B_m — a fraction in [0, 1]
Problem: division by zero when B_m = 0. A metric that is already perfect has no gap to close, but the formula is undefined rather than vacuous, and a satellite could declare an already-clean metric as a target to harvest easy verdicts.
Options:
- Excluded (recommended). A metric with
B_m = 0 is ineligible as a target; declaring it yields reject_null. It is still evaluated — as collateral, where D_c = A_c - B_c needs no division and catches any regression from zero. No vacuous accepts, no formula change.
- Vacuous accept: define
G_m = 1 when B_m = A_m = 0. Gameable (see problem above).
- Undefined-is-fatal: the evaluation is invalid if any target has
B_m = 0. Stricter than 1 but turns a data condition into a tooling error.
(b) capability_ok: aggregate pass-rate vs per-probe identity
Current wording (lines 89–94): the table defines capability_ok as P_after >= P_before - tolerance (an aggregate rate), but the prose says:
Provisional tolerance: one probe, i.e. 1/N for a suite of N probes. A satellite that fails two or more probes its baseline passed is rejected regardless of metric improvement.
Problem: the two definitions disagree. A satellite that breaks two baseline-passing probes while fixing two baseline-failing ones keeps the aggregate rate identical (capability_ok true by the table) yet fails the prose test. The paper analogue the appendix cites (line 570: "The first maps to capability_ok") is a per-benchmark gate — rejected if any capability benchmark degrades decisively — which supports identity tracking.
Options:
- Per-probe identity (recommended).
capability_ok ⇔ at most one probe that passed at baseline fails after (tolerance = one probe, expressed against loaded suite size). Newly-passing probes are recorded but cannot buy back regressions. The capability block gains a probes_regressed list naming any such probe. Table and prose then agree.
- Aggregate rate: keep the table formula, delete the "two or more probes" prose sentence.
- Both gates must pass (aggregate AND identity) — strictly stronger than 1 with little extra discrimination.
(c) Multi-target acceptance rule
Current wording: the verdict table (lines 108–138) is written in the singular — accept: "Target metric improved; …"; reject_null: "Target metric unchanged or worse" — but the v2 example (lines 229–248) declares two targets (LPS gap_closed 0.585, TII 0.333).
Problem: with multiple targets it is undefined whether accept requires all targets to improve, any, or per-target gap thresholds.
Options:
- All-targets (recommended). Every declared target must improve (
G_m > 0); if any target is unchanged or worse → reject_null. Declaring a target is a claim; this keeps target lists honest (no padding), adds no new constants, and a metric the satellite is unsure of simply stays out of target_metrics, where the collateral thresholds still protect it.
- Any-target: at least one target improved. Non-improving targets sit in neither the accept condition nor the collateral set — they become unprotected.
- Per-target minimum gap thresholds: each target must close at least some fraction. Needs a new provisional constant and a calibration story.
(d) frontier_record plurality for multi-target satellites
Current wording: efficacy-v2 example, line 271: "frontier_record": "frontier/LPS-2026-09-01.json" — a single string. vexometer-frontier-v1 (line 306) is per-metric ("metric": "LPS"), and its monotone-frontier invariant (line 360) plus the ten-small-multiples dashboard both assume one metric per record.
Problem: the example's own satellite has two targets; a single string cannot reference the TII search history.
Options:
- Pluralise the field (recommended). v2.1 renames it
frontier_records, an array with one per-metric record per target (["frontier/LPS-…", "frontier/TII-…"]). frontier-v1 itself is untouched — the per-metric invariant stays clean. Since no emitter exists yet (D5 tooling is unbuilt), migration cost today is zero.
- Multi-metric frontier records: widen frontier-v1 to hold several metrics — breaks the per-metric monotone invariant and complicates the frontier plot.
- Single string to the "primary" target's record — silently discards the other target's search history.
(e) v1→v2 lifting: spec it or drop the promise
Current wording (lines 288–290):
A v1 report can be mechanically lifted to v2 shape, but it cannot be assigned a verdict … Lifted reports carry "verdict": "unverified".
Problem: "mechanically lifted" is promised but unspecified — where baseline/after come from when v1's metrics lacked them, what fills scenario_set, whether lifting is an algorithm or ad hoc. Meanwhile zero satellites exist, so zero v1 reports exist: the lifting machinery currently has no customer.
Options:
- Drop the promise; v2-only emission (recommended). v2.1 keeps the field-correspondence table as documentation, and replaces the lifting sentence with: a v1 report is never converted — to obtain a verdict, re-evaluate under v2. Honest, zero machinery, nothing existing is orphaned.
- Write a normative lifting appendix (absent v1 data maps to explicit null, never synthesised;
lifted_from marker; verdict: "unverified") plus an efficacy-lift subcommand in the D5 tool.
- Leave the prose as-is — rejected by the review finding.
(f) Held-out scenario set — authorization to build before any satellite work
This one is not a wording choice. The appendix already states the rule and the gap (lines 571–574):
a satellite must never be tuned against the scenario set it is scored on — the content-addressed scenario_set makes that auditable, though a held-out scenario set does not yet exist and is future work.
The sitrep's route forward (item 4) orders this before the first satellite (D6), because once tuning has started against the only scenario set, a held-out split can never be recovered — it is too late by definition.
Asking for: authorization to stand up the held-out set now — split the scenario corpus into tuning and held-out partitions, record both content-addressed hashes in-repo, score only against the held-out hash, and promote the rule from the appendix NOTE into the normative v2.1 text.
Options:
- Authorized (recommended) — build it as part of the v2.1/D5 work, before D6 begins.
- Deferred to first satellite proposal — accepts the risk that tuning discipline cannot be retro-imposed.
- Authorized with owner-specified partition method or corpus source (say which).
Filed from job 22d3a88e per docs/sitrep-2026-09-01.adoc route-forward item 1. Done-condition for D1: rulings recorded here, protocol amended to v2.1 with examples updated.
Six normative questions were deferred during the review of PR #63 (efficacy protocol, CodeRabbit findings 1, 4, 5, 8, 9, 11 — see the review-disposition comment there). The sitrep (
docs/sitrep-2026-09-01.adoc, debt item D1) batches them into this one issue per the 2026-08-27 batching ruling.How to rule: six one-line replies resolve the whole issue, e.g.
a: 1, b: 1, c: 1, d: 1, e: 1, f: authorized. Anything not option 1 with a note is equally fine. After rulings, one PR amendsvexometer/docs/EFFICACY-PROTOCOL.adocto v2.1 with all examples updated — no normative text changes before then.Live consequence while open: the D5
efficacy-reporttool (in progress) will refuse to compute a zero-baselinegap_closedwith an explicitawaiting ruling D1aerror rather than silently picking a semantic.All line references are to
vexometer/docs/EFFICACY-PROTOCOL.adocatbf2ec6d.(a)
G_mwhen the baseline is zeroCurrent wording (lines 40–47):
Problem: division by zero when
B_m = 0. A metric that is already perfect has no gap to close, but the formula is undefined rather than vacuous, and a satellite could declare an already-clean metric as a target to harvest easy verdicts.Options:
B_m = 0is ineligible as a target; declaring it yieldsreject_null. It is still evaluated — as collateral, whereD_c = A_c - B_cneeds no division and catches any regression from zero. No vacuous accepts, no formula change.G_m = 1whenB_m = A_m = 0. Gameable (see problem above).B_m = 0. Stricter than 1 but turns a data condition into a tooling error.(b)
capability_ok: aggregate pass-rate vs per-probe identityCurrent wording (lines 89–94): the table defines
capability_okasP_after >= P_before - tolerance(an aggregate rate), but the prose says:Problem: the two definitions disagree. A satellite that breaks two baseline-passing probes while fixing two baseline-failing ones keeps the aggregate rate identical (
capability_oktrue by the table) yet fails the prose test. The paper analogue the appendix cites (line 570: "The first maps tocapability_ok") is a per-benchmark gate — rejected if any capability benchmark degrades decisively — which supports identity tracking.Options:
capability_ok⇔ at most one probe that passed at baseline fails after (tolerance = one probe, expressed against loaded suite size). Newly-passing probes are recorded but cannot buy back regressions. Thecapabilityblock gains aprobes_regressedlist naming any such probe. Table and prose then agree.(c) Multi-target acceptance rule
Current wording: the verdict table (lines 108–138) is written in the singular —
accept: "Target metric improved; …";reject_null: "Target metric unchanged or worse" — but the v2 example (lines 229–248) declares two targets (LPSgap_closed0.585, TII 0.333).Problem: with multiple targets it is undefined whether
acceptrequires all targets to improve, any, or per-target gap thresholds.Options:
G_m > 0); if any target is unchanged or worse →reject_null. Declaring a target is a claim; this keeps target lists honest (no padding), adds no new constants, and a metric the satellite is unsure of simply stays out oftarget_metrics, where the collateral thresholds still protect it.(d)
frontier_recordplurality for multi-target satellitesCurrent wording: efficacy-v2 example, line 271:
"frontier_record": "frontier/LPS-2026-09-01.json"— a single string.vexometer-frontier-v1(line 306) is per-metric ("metric": "LPS"), and its monotone-frontier invariant (line 360) plus the ten-small-multiples dashboard both assume one metric per record.Problem: the example's own satellite has two targets; a single string cannot reference the TII search history.
Options:
frontier_records, an array with one per-metric record per target (["frontier/LPS-…", "frontier/TII-…"]).frontier-v1itself is untouched — the per-metric invariant stays clean. Since no emitter exists yet (D5 tooling is unbuilt), migration cost today is zero.(e) v1→v2 lifting: spec it or drop the promise
Current wording (lines 288–290):
Problem: "mechanically lifted" is promised but unspecified — where
baseline/aftercome from when v1'smetricslacked them, what fillsscenario_set, whether lifting is an algorithm or ad hoc. Meanwhile zero satellites exist, so zero v1 reports exist: the lifting machinery currently has no customer.Options:
lifted_frommarker;verdict: "unverified") plus anefficacy-liftsubcommand in the D5 tool.(f) Held-out scenario set — authorization to build before any satellite work
This one is not a wording choice. The appendix already states the rule and the gap (lines 571–574):
The sitrep's route forward (item 4) orders this before the first satellite (D6), because once tuning has started against the only scenario set, a held-out split can never be recovered — it is too late by definition.
Asking for: authorization to stand up the held-out set now — split the scenario corpus into tuning and held-out partitions, record both content-addressed hashes in-repo, score only against the held-out hash, and promote the rule from the appendix NOTE into the normative v2.1 text.
Options:
Filed from job 22d3a88e per
docs/sitrep-2026-09-01.adocroute-forward item 1. Done-condition for D1: rulings recorded here, protocol amended to v2.1 with examples updated.