Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 31 additions & 2 deletions Justfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@
# Default recipe: list available commands
import? "contractile.just"

# Pass recipe arguments through as shell positionals ("$@") so quoting
# survives; the per-recipe [positional-arguments] attribute needs just >= 1.29,
# but CI installs just from apt (ubuntu-latest ships 1.21) and an unknown
# attribute is a parse error that kills every recipe.
set positional-arguments := true

default:
@just --list

Expand All @@ -20,8 +26,12 @@ build-vext:
build-lazy-eliminator:
cd lazy-eliminator && just build

# Build the efficacy evaluator (Rust)
build-efficacy:
cd vexometer-efficacy && cargo build --release

# Build all components
build-all: build-vexometer build-vext build-lazy-eliminator
build-all: build-vexometer build-vext build-lazy-eliminator build-efficacy

# Run vexometer tests
test-vexometer:
Expand All @@ -39,30 +49,49 @@ test-vext:
test-lazy-eliminator:
cd lazy-eliminator && just test

# Run efficacy-evaluator tests (protocol examples are the fixtures)
test-efficacy:
cd vexometer-efficacy && (cargo test --offline || cargo test)

# vext-email-gateway status check
test-vext-email-gateway:
@echo "vext-email-gateway is currently prototype-stage and not part of the required test-all gate."
@echo "See vext-email-gateway/README.adoc and ROADMAP.adoc for current wiring status."

# Run all tests
test-all: test-vexometer test-vext test-lazy-eliminator
test-all: test-vexometer test-vext test-lazy-eliminator test-efficacy

# Evaluate a satellite run and emit a vexometer-efficacy-v2 report
efficacy-report *ARGS:
cd vexometer-efficacy && cargo run --release --quiet -- report "$@"

# Record a search attempt in a vexometer-frontier-v1 record
efficacy-attempt *ARGS:
cd vexometer-efficacy && cargo run --release --quiet -- attempt "$@"

# Validate efficacy reports and frontier records by recomputation
efficacy-validate *ARGS:
cd vexometer-efficacy && cargo run --release --quiet -- validate "$@"

# Run benchmark suites
bench-all: bench-vexometer

# Clean all build artifacts
clean:
cd vext && cargo clean
cd vexometer-efficacy && cargo clean
cd vexometer && just clean || true
cd lazy-eliminator && just clean || true

# Check formatting across Rust components
fmt-check:
cd vext && cargo fmt -- --check
cd vexometer-efficacy && cargo fmt -- --check

# Run clippy on Rust components
lint:
cd vext && cargo clippy -- -D warnings
cd vexometer-efficacy && cargo clippy --all-targets -- -D warnings

# Run contractiles Mustfile invariants across all components
must-all:
Expand Down
1 change: 1 addition & 0 deletions scripts/run-must-gates.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ components=(
"vext-email-gateway"
"vexometer-satellites"
"lazy-eliminator"
"vexometer-efficacy"
"satellite-template"
)

Expand Down
1 change: 1 addition & 0 deletions scripts/trust/generate-manifest.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ else
"vext-email-gateway"
"vexometer-satellites"
"lazy-eliminator"
"vexometer-efficacy"
"satellite-template"
)
fi
Expand Down
1 change: 1 addition & 0 deletions scripts/trust/verify-manifest.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ else
"vext-email-gateway"
"vexometer-satellites"
"lazy-eliminator"
"vexometer-efficacy"
"satellite-template"
)
fi
Expand Down
1 change: 1 addition & 0 deletions vexometer-efficacy/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/target
8 changes: 8 additions & 0 deletions vexometer-efficacy/.trust/trust-manifest.sha256
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# trust-manifest v1
# component=vexometer-efficacy
# generated_at=2026-09-01T15:55:49Z
3ea7341c2a55bea766ffa7c34879168001f701982c3427c8f3ff874b4b907c3e README.adoc
99bf8c708656fee9beba0c4812aac55a6fd3b4fdaaa989a9b6a13b7dc3c4b5ba ROADMAP.adoc
b1245e468709a6c75e530412da6480943bf53c836df0ca108aaf39843886e6cb SECURITY.adoc
9c80ff2e60fdb772a0479b46b140e0ce08e4e37bc39e6d7e257aa3d5d1281d18 contractiles/must/Mustfile
3ac4606620454d844d8f0d0580fe32072a8a0b6821c93a74df64c3ed597e3640 contractiles/trust/Trustfile.a2ml
107 changes: 107 additions & 0 deletions vexometer-efficacy/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

15 changes: 15 additions & 0 deletions vexometer-efficacy/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# SPDX-License-Identifier: MPL-2.0
[package]
name = "vexometer-efficacy"
version = "0.1.0"
edition = "2021"
license = "MPL-2.0"
description = "Efficacy evaluator and frontier-record writer for the vexometer ISA efficacy protocol (vexometer-efficacy-v2 / vexometer-frontier-v1)"
repository = "https://github.com/hyperpolymath/vexometer"

[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"

[profile.release]
lto = true
139 changes: 139 additions & 0 deletions vexometer-efficacy/README.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
// SPDX-License-Identifier: CC-BY-SA-4.0
= vexometer-efficacy
:toc:

Efficacy evaluator and frontier-record writer for the vexometer ISA
efficacy protocol. This tool is the executable half of
link:../vexometer/docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc]:
it computes `G_m`, collateral deltas, `D_ISA`, and the capability proxy,
applies the six-verdict acceptance rule with its precedence order, emits
`vexometer-efficacy-v2` reports, maintains `vexometer-frontier-v1`
records under the monotone-frontier invariant, and validates both
document shapes by recomputing every derived number.

== Design rule: refuse where the protocol is undecided

Six normative questions are open in
https://github.com/hyperpolymath/vexometer/issues/69[issue #69] (debt
item D1). Where one of them bites, this tool *refuses with an explicit
error* naming the question rather than silently picking a semantic:

[cols="1,4,2",options="header"]
|===
|Question |When it bites |Behaviour

|D1a
|A declared target metric has baseline `B_m = 0` (division by zero in
`G_m`)
|Hard refusal, exit code 2

|D1b
|Per-probe results are supplied for both measurements and the aggregate
pass-rate gate disagrees with the per-probe identity gate
|Hard refusal, exit code 2

|D1c
|Multiple targets are declared and some improved while others did not
|Hard refusal, exit code 2

|D1d
|Multiple targets with the singular `frontier_record` field
|Report is emitted, with a warning on stderr

|D1e
|v1→v2 lifting
|Unimplemented — no `lift` subcommand exists
|===

After the rulings land and the protocol is amended to v2.1, these
refusals are replaced by the ruled semantics.

== The protocol's examples are the test fixtures

The integration tests read `../vexometer/docs/EFFICACY-PROTOCOL.adoc`
at build time, extract its example JSON blocks, and require that the
validator accepts both and that the evaluator reproduces the efficacy
example value-for-value from raw inputs (including `D_ISA = -2.71`
under the default category weights in
link:../vexometer/docs/METRICS.adoc[METRICS.adoc]). If the protocol and
this implementation drift apart, `cargo test` fails loudly.

== CLI

[source,console]
----
$ vexometer-efficacy report --baseline baseline.json --after after.json \
--targets LPS,TII --satellite vex-verbosity-compressor \
--sample-size 500 --output report.json \
[--methodology "A/B testing with vexometer validation"] \
[--notes "..."] [--frontier-record frontier/LPS-....json] \
[--traces-available true|false] [--date YYYY-MM-DD] [--scenario-set SHA]

$ vexometer-efficacy attempt --frontier frontier/LPS-2026-09-01.json \
--baseline baseline.json --after after.json --targets LPS --metric LPS \
--satellite vex-verbosity-compressor --config config-042 \
[--model-profile STR] [--timestamp ISO8601] [--scenario-set SHA] \
[--baseline-isa 4.63] # required when creating a new frontier record

$ vexometer-efficacy validate report.json frontier.json ...
----

Bare `validate` arguments are routed by each document's own `version`
field; `--efficacy FILE` / `--frontier FILE` force a kind when a
document lacks one. The same commands are exposed at the monorepo root
as `just efficacy-report`, `just efficacy-attempt`, and
`just efficacy-validate`.

Exit codes: `0` success (any verdict, including rejections — a computed
rejection is a successful evaluation), `1` usage or data error, `2` open
D1 ruling required, `3` validation failed.

== Measurement input format

Both `--baseline` and `--after` take a JSON document of one measurement
pass over one content-addressed scenario set:

[source,json]
----
{
"scenario_set": "sha256:6b2f...",
"metrics": {
"LPS": 0.41,
"TII": { "score": 0.33, "std_dev": 0.07, "confidence": 0.95, "p_value": 0.004 },
"EFR": 0.19, "PQ": 0.28, "TAI": 0.15, "ICS": 0.22,
"CII": 0.31, "SRS": 0.26, "SFR": 0.24, "RCI": 0.30
},
"probes": {
"total": 13,
"passed": 12,
"results": { "PROBE-CAPABILITY-001": true, "...": true }
}
}
----

* `metrics` must cover exactly the ten ISA metrics. A bare number and a
`{score, std_dev, confidence, p_value}` object are both accepted;
statistics are carried into the report when present.
* `probes.results` (per-probe outcomes) is optional; when both
measurements carry it, the per-probe identity gate is cross-checked
against the aggregate gate (see D1b above).
* `scenario_set` must match between baseline and after — tuning against
a different set than you score on is exactly what the protocol's
audit trail exists to catch.

== Building and testing

From this directory (or via the monorepo Justfile):

[source,console]
----
$ cargo build --release # or: just build-efficacy
$ cargo test # or: just test-efficacy
----

The only dependencies are `serde` and `serde_json`; `Cargo.lock` is
committed.

== Licence

Code MPL-2.0, documentation CC-BY-SA-4.0, per repository policy.
Loading
Loading