Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Justfile
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,10 @@ efficacy-report *ARGS:
efficacy-attempt *ARGS:
cd vexometer-efficacy && cargo run --release --quiet -- attempt "$@"

# Mechanically lift a vexometer-efficacy-v1 report to v2.1 shape (ruling e2)
efficacy-lift *ARGS:
cd vexometer-efficacy && cargo run --release --quiet -- lift "$@"

# Validate efficacy reports and frontier records by recomputation
efficacy-validate *ARGS:
cd vexometer-efficacy && cargo run --release --quiet -- validate "$@"
Expand Down
2 changes: 1 addition & 1 deletion lazy-eliminator/.trust/trust-manifest.sha256
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# trust-manifest v1
# component=lazy-eliminator
# generated_at=2026-09-01T14:22:53Z
# generated_at=2026-09-01T23:29:45Z
339d25795fa89149354d4101c533492f0e5bbe39fc953ac02be17a225f2fd270 README.adoc
6772e621da4e50257728886f568bd652b8e9c58a02aec325112e57def73da46a ROADMAP.adoc
504199ed09a9acbd183fe9c37a8330ec254f33f5612e0f36f2f5e7be170a57ff SECURITY.adoc
Expand Down
2 changes: 1 addition & 1 deletion satellite-template/.trust/trust-manifest.sha256
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# trust-manifest v1
# component=satellite-template
# generated_at=2026-09-01T14:22:53Z
# generated_at=2026-09-01T23:29:45Z
e90437cd512f3ac6824b42e733394bc69dfa4bcab046d779842bc876e11f870e README.adoc
5a62f5611eecfafa43d931b4d7e0917fa1b0f0275fdfd74c61415ab635e72805 ROADMAP.adoc
38ccfdc1a04c12616acfb030522358383702184480f52507a5f72fccccbe76b9 SECURITY.adoc
Expand Down
6 changes: 3 additions & 3 deletions vexometer-efficacy/.trust/trust-manifest.sha256
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# trust-manifest v1
# component=vexometer-efficacy
# generated_at=2026-09-01T15:55:49Z
3ea7341c2a55bea766ffa7c34879168001f701982c3427c8f3ff874b4b907c3e README.adoc
99bf8c708656fee9beba0c4812aac55a6fd3b4fdaaa989a9b6a13b7dc3c4b5ba ROADMAP.adoc
# generated_at=2026-09-01T23:29:45Z
df1c5e511fc5e2e7cbcd094fea57728ce193ee267a267037f4043ae5edbf3279 README.adoc
272d7670bb46837c46db61d519d849c60314fecaaf3cd807ee9fdd18ec76ba35 ROADMAP.adoc
b1245e468709a6c75e530412da6480943bf53c836df0ca108aaf39843886e6cb SECURITY.adoc
9c80ff2e60fdb772a0479b46b140e0ce08e4e37bc39e6d7e257aa3d5d1281d18 contractiles/must/Mustfile
3ac4606620454d844d8f0d0580fe32072a8a0b6821c93a74df64c3ed597e3640 contractiles/trust/Trustfile.a2ml
111 changes: 66 additions & 45 deletions vexometer-efficacy/README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,52 +11,65 @@ applies the six-verdict acceptance rule with its precedence order, emits
records under the monotone-frontier invariant, and validates both
document shapes by recomputing every derived number.

== Design rule: refuse where the protocol is undecided
== The ruled semantics (protocol v2.1)

Six normative questions are open in
https://github.com/hyperpolymath/vexometer/issues/69[issue #69] (debt
item D1). Where one of them bites, this tool *refuses with an explicit
error* naming the question rather than silently picking a semantic:
The six normative questions this tool originally refused to guess at
(https://github.com/hyperpolymath/vexometer/issues/69[issue #69],
D1a-D1f) were ruled `a1, b1, c1, d1, e2, f1` on 2026-09-01 and are
implemented here:

[cols="1,4,2",options="header"]
[cols="1,4,3",options="header"]
|===
|Question |When it bites |Behaviour

|D1a
|A declared target metric has baseline `B_m = 0` (division by zero in
`G_m`)
|Hard refusal, exit code 2

|D1b
|Per-probe results are supplied for both measurements and the aggregate
pass-rate gate disagrees with the per-probe identity gate
|Hard refusal, exit code 2

|D1c
|Multiple targets are declared and some improved while others did not
|Hard refusal, exit code 2

|D1d
|Multiple targets with the singular `frontier_record` field
|Report is emitted, with a warning on stderr

|D1e
|Ruling |When it bites |Behaviour

|a1
|A declared target metric has baseline `B_m = 0`
|`G_m := 0`, so the target cannot improve and the verdict is
`reject_null`; the report lists the metric in a diagnosability warning.
Zero-baseline *collateral* metrics stay fully protected.

|b1
|Per-probe results exist in both measurements
|The per-probe identity gate is normative: at most one baseline-passing
probe may fail after; newly-passing probes buy nothing back. The
aggregate pass-rate gate is the degraded fallback when per-probe results
are absent.

|c1
|Multiple targets are declared
|All-targets rule: every declared target must improve, or the verdict is
`reject_null`.

|d1
|Frontier references
|Plural `frontier_records`, one per-metric record per target, in target
order; a length mismatch is a hard error and the pre-ruling singular
`frontier_record` key fails validation.

|e2
|v1→v2 lifting
|Unimplemented — no `lift` subcommand exists
|The `lift` subcommand: v1 fields carried verbatim, missing v2 evidence
as explicit `null`, `lifted_from` marker, `verdict: "unverified"`
(reserved for lifted reports).

|f1
|Scenario-set provenance
|`validate --scenario-registry FILE` checks every scored `scenario_set`
against the held-out partition registry at
`../vexometer/data/scenario_sets/registry.json`.
|===

After the rulings land and the protocol is amended to v2.1, these
refusals are replaced by the ruled semantics.

== The protocol's examples are the test fixtures

The integration tests read `../vexometer/docs/EFFICACY-PROTOCOL.adoc`
at build time, extract its example JSON blocks, and require that the
validator accepts both and that the evaluator reproduces the efficacy
validator accepts them all, that the evaluator reproduces the efficacy
example value-for-value from raw inputs (including `D_ISA = -2.71`
under the default category weights in
link:../vexometer/docs/METRICS.adoc[METRICS.adoc]). If the protocol and
this implementation drift apart, `cargo test` fails loudly.
link:../vexometer/docs/METRICS.adoc[METRICS.adoc]), and that `lift`
reproduces the protocol's lifted example from its v1 example. If the
protocol and this implementation drift apart, `cargo test` fails
loudly.

== CLI

Expand All @@ -66,7 +79,8 @@ $ vexometer-efficacy report --baseline baseline.json --after after.json \
--targets LPS,TII --satellite vex-verbosity-compressor \
--sample-size 500 --output report.json \
[--methodology "A/B testing with vexometer validation"] \
[--notes "..."] [--frontier-record frontier/LPS-....json] \
[--notes "..."] \
[--frontier-records frontier/LPS-....json]... \
[--traces-available true|false] [--date YYYY-MM-DD] [--scenario-set SHA]

$ vexometer-efficacy attempt --frontier frontier/LPS-2026-09-01.json \
Expand All @@ -75,18 +89,24 @@ $ vexometer-efficacy attempt --frontier frontier/LPS-2026-09-01.json \
[--model-profile STR] [--timestamp ISO8601] [--scenario-set SHA] \
[--baseline-isa 4.63] # required when creating a new frontier record

$ vexometer-efficacy validate report.json frontier.json ...
$ vexometer-efficacy lift --input v1-report.json --output lifted.json

$ vexometer-efficacy validate report.json frontier.json ... \
[--scenario-registry ../vexometer/data/scenario_sets/registry.json]
----

Bare `validate` arguments are routed by each document's own `version`
field; `--efficacy FILE` / `--frontier FILE` force a kind when a
document lacks one. The same commands are exposed at the monorepo root
as `just efficacy-report`, `just efficacy-attempt`, and
`just efficacy-validate`.
Pass `--frontier-records` once per target metric, in target order
(ruling d1). Bare `validate` arguments are routed by each document's own
`version` field; `--efficacy FILE` / `--frontier FILE` force a kind when
a document lacks one, and `--scenario-registry` enforces ruling f1
against every scored scenario set. The same commands are exposed at the
monorepo root as `just efficacy-report`, `just efficacy-attempt`,
`just efficacy-lift`, and `just efficacy-validate`.

Exit codes: `0` success (any verdict, including rejections — a computed
rejection is a successful evaluation), `1` usage or data error, `2` open
D1 ruling required, `3` validation failed.
rejection is a successful evaluation), `1` usage or data error, `3`
validation failed. (Exit code `2`, the pre-ruling "open D1 question"
refusal, is retired.)

== Measurement input format

Expand Down Expand Up @@ -115,8 +135,9 @@ pass over one content-addressed scenario set:
`{score, std_dev, confidence, p_value}` object are both accepted;
statistics are carried into the report when present.
* `probes.results` (per-probe outcomes) is optional; when both
measurements carry it, the per-probe identity gate is cross-checked
against the aggregate gate (see D1b above).
measurements carry it, the per-probe identity gate is normative
(ruling b1) and the report records any regressed probes in
`capability.probes_regressed`.
* `scenario_set` must match between baseline and after — tuning against
a different set than you score on is exactly what the protocol's
audit trail exists to catch.
Expand Down
14 changes: 8 additions & 6 deletions vexometer-efficacy/ROADMAP.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,15 @@
* [x] Protocol examples as live test fixtures (drift fails `cargo test`)
* [x] Explicit refusals on open D1 questions (issue #69)

== After the D1 rulings (v0.2, blocked on issue #69)
== The D1 rulings (v0.2, shipped — issue #69 ruled `a1,b1,c1,d1,e2,f1`)

* [ ] Replace each D1a–D1d refusal with the ruled semantic
* [ ] `frontier_records` plurality per ruling (d)
* [ ] v1→v2 lifting: implement or formally drop per ruling (e)
* [ ] Held-out scenario-set support per ruling (f)
* [ ] Track the protocol's v2.1 text (same PR as the amendment)
* [x] Replace each D1a–D1d refusal with the ruled semantic (a1 zero
baseline, b1 per-probe identity gate, c1 all-targets rule)
* [x] `frontier_records` plurality per ruling d1
* [x] v1→v2 lifting: `lift` subcommand per ruling e2
* [x] Held-out scenario-set registry + `--scenario-registry` per
ruling f1
* [x] Track the protocol's v2.1 text (same PR as the amendment)

== Later

Expand Down
Loading
Loading