From 02978d9ccd95f5b43f7b59331dbbaca71bf00bf8 Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" <6759885+hyperpolymath@users.noreply.github.com> Date: Tue, 1 Sep 2026 03:57:24 +0100 Subject: [PATCH 1/6] docs(vexometer): add ISA efficacy protocol with cross-metric acceptance rule Adds docs/EFFICACY-PROTOCOL.adoc, which closes a gap in the satellite architecture: ISA claims the role of validating satellite efficacy claims, but vexometer-efficacy-v1 reports only target-metric reductions. A satellite that lowers its target metric while raising another, or while degrading capability, has moved the irritation surface rather than shrunk it, and nothing in the existing protocol can say so. The document specifies: - An acceptance rule with six verdicts (accept, accept_with_warning, and four reject reasons), gated on collateral deltas across all ten metrics, a capability proxy, and a weighted net ISA delta computed from the category weights in METRICS.adoc. Thresholds are stated as provisional constants pending calibration. - A capability proxy defined as pass-rate over the existing 13-probe behavioural suite, reusing data/probes/behavioural_probes.json rather than introducing new machinery. - A collateral risk register naming the predicted regression direction for each of the twelve planned satellites, including three cases where the risk lands on the satellite's own target metric because SFR and TII penalise both excess and deficit. - vexometer-efficacy-v2, strictly additive to v1, adding collateral_metrics, capability, isa_delta and verdict. Lifted v1 reports carry verdict "unverified". - vexometer-frontier-v1, an ordered record of every configuration tried against a metric including rejections, with a monotonic best-so-far frontier. - Search budget tiers, marked as a hypothesis for the frontier records to test rather than an imported result. - A scope boundary stating plainly that this is an irritation-surface protocol and not a safety evaluation. Proposed Justfile recipes are marked as not yet implemented, alongside the existing aspirational recipes in SATELLITES.adoc. Cross-referenced from README.adoc, METRICS.adoc and SATELLITES.adoc. Co-Authored-By: Claude Opus 5 --- vexometer/README.adoc | 2 + vexometer/docs/EFFICACY-PROTOCOL.adoc | 546 ++++++++++++++++++++++++++ vexometer/docs/METRICS.adoc | 1 + vexometer/docs/SATELLITES.adoc | 1 + 4 files changed, 550 insertions(+) create mode 100644 vexometer/docs/EFFICACY-PROTOCOL.adoc diff --git a/vexometer/README.adoc b/vexometer/README.adoc index ab96121..b68f79b 100644 --- a/vexometer/README.adoc +++ b/vexometer/README.adoc @@ -196,6 +196,7 @@ vexometer/ | +-- SPECIFICATION.md # Full technical specification | +-- METRICS.adoc # All 10 metrics detailed | +-- SATELLITES.adoc # Intervention satellite architecture +| +-- EFFICACY-PROTOCOL.adoc # Satellite acceptance and frontier records | +-- letter_lmsys_arena.md # LMSYS Arena proposal +-- alire.toml # Alire package manifest +-- vexometer.gpr # GNAT project file @@ -405,6 +406,7 @@ Priority areas: * link:docs/SPECIFICATION.md[SPECIFICATION.md] - Full technical specification * link:docs/METRICS.adoc[METRICS.adoc] - Detailed metric reference * link:docs/SATELLITES.adoc[SATELLITES.adoc] - Satellite architecture +* link:docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc] - How satellite efficacy claims are accepted or rejected * link:CLAUDE.md[CLAUDE.md] - AI assistant guidance == License diff --git a/vexometer/docs/EFFICACY-PROTOCOL.adoc b/vexometer/docs/EFFICACY-PROTOCOL.adoc new file mode 100644 index 0000000..935fe9d --- /dev/null +++ b/vexometer/docs/EFFICACY-PROTOCOL.adoc @@ -0,0 +1,546 @@ +// SPDX-License-Identifier: CC-BY-SA-4.0 +// Copyright (c) Jonathan D.A. Jewell +// SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell + += ISA Efficacy Protocol +:toc: left +:toclevels: 3 +:icons: font +:source-highlighter: rouge + +== Overview + +This document defines how ISA decides whether a satellite intervention +*works*. It extends the efficacy machinery in link:SATELLITES.adoc[SATELLITES.adoc] +with two things that machinery currently lacks: + +. An *acceptance rule*. A satellite that reduces its target metric while + raising another metric, or while degrading capability, has not improved + the irritation surface -- it has moved it. `vexometer-efficacy-v1` + reports only target-metric reductions and so cannot express this. +. A *search record*. Efficacy is currently a single before/after snapshot. + In practice a satellite is the survivor of many attempted configurations, + most of which fail. Recording the ordered search -- including the + rejections -- is what makes an efficacy claim auditable. + +Technical compatibility note: metric package and protocol identifiers +continue to use the `vexometer` prefix. + +== Definitions + +*Baseline* (`B_m`):: +The ISA score for metric `m` on unmodified model output, over a fixed +scenario set. Metrics are normalised 0-1, lower is better +(link:METRICS.adoc[METRICS.adoc]). + +*Post-intervention* (`A_m`):: +The ISA score for metric `m` on the same scenario set, processed through +the satellite under evaluation. + +*Gap closed* (`G_m`):: +The fraction of the available irritation gap removed. Since the floor is +0, the available gap is the baseline itself: + +[source] +---- +G_m = (B_m - A_m) / B_m expressed as a percentage +---- + +*Collateral delta* (`D_c`):: +For every metric `c` other than the target(s), the absolute change +`A_c - B_c`. Positive means the metric got worse. + +*Net ISA delta* (`D_ISA`):: +The weighted change in the overall ISA score, using the category weights +in link:METRICS.adoc[METRICS.adoc]: + +[source] +---- +D_ISA = ( SUM(w_c * (A_c - B_c)) / SUM(w_c) ) * 100 +---- + +A satellite must produce `D_ISA < 0` to be accepted. This is the rule +that prevents a large reduction in a low-weight metric from paying for a +small regression in a high-weight one. + +== The Capability Proxy + +An intervention that suppresses irritation by suppressing usefulness is +not an improvement. ISA does not need new machinery to detect this: the +existing behavioural probe suite (`data/probes/behavioural_probes.json`, 13 +standardised probes) already exercises brevity, competence, sycophancy, +constraint, uncertainty, and direct-answer behaviours. + +*Capability proxy*: the probe pass-rate over the 13-probe suite, measured +on baseline output and on satellite-processed output. + +[cols="2,3", options="header"] +|=== +|Term |Definition + +|`P_before` |Probe pass-rate on unmodified output +|`P_after` |Probe pass-rate on satellite-processed output +|`capability_ok` |`P_after >= P_before - tolerance` +|=== + +*Provisional tolerance*: one probe (`1/13 ~= 0.077`). A satellite that +fails two or more probes its baseline passed is rejected regardless of +metric improvement. + +NOTE: The probe suite is a proxy, not a capability benchmark. It detects +gross degradation, not subtle loss. Satellites whose interventions are +capability-adjacent (verbosity compression, scope governance) should +supply an additional domain benchmark alongside the probe result. + +== The Acceptance Rule + +A satellite evaluation yields exactly one verdict. + +[cols="2,4,1", options="header"] +|=== +|Verdict |Condition |Frontier + +|`accept` +|Target metric improved; every collateral delta `<= 0.02`; +`capability_ok`; `D_ISA < 0` +|Advances + +|`accept_with_warning` +|As `accept`, but some collateral delta falls in `(0.02, 0.05]`. The +regressed metric must be named in the efficacy report and in the +satellite README. +|Advances + +|`reject_collateral` +|Any collateral delta `> 0.05` +|Does not advance + +|`reject_capability` +|`capability_ok` is false +|Does not advance + +|`reject_net` +|`D_ISA >= 0` -- the weighted irritation surface did not shrink +|Does not advance + +|`reject_null` +|Target metric unchanged or worse +|Does not advance +|=== + +*Provisional constants*: the `0.02` / `0.05` collateral thresholds and the +one-probe capability tolerance are stated so that evaluations are +reproducible today. They are expected to be recalibrated once enough +frontier records exist to characterise measurement noise per metric. +Until then, report the raw deltas alongside the verdict so that a +recalibration can be applied retrospectively. + +=== Collateral Risk Register + +Every satellite has a predicted collateral direction. Evaluations should +measure all ten metrics, but these pairings are where regressions are +most likely and must always be reported explicitly. + +[cols="2,1,3", options="header"] +|=== +|Satellite |Targets |Predicted collateral risk + +|vex-lazy-eliminator |CII, LPS +|↑ TII -- completeness enforcement lengthens output + +|vex-hallucination-guard |EFR +|↑ TII (verification latency); ↑ PQ (verification framed as warnings) + +|vex-sycophancy-shield |LPS, EFR +|↑ PQ -- bluntness is read as lecturing + +|vex-confidence-calibrator |EFR +|↑ LPS -- calibrated uncertainty language *is* hedge-word density, an +explicit LPS component + +|vex-specification-anchor |SFR, ICS +|↑ SRS -- an immutable spec ledger rewards defending the original approach + +|vex-instruction-persistence |TII, ICS +|↑ SRS -- instruction salience shades into approach anchoring + +|vex-backtrack-enabler |SRS, ICS +|↑ SFR (restarts mutate scope); ↑ TII (restart churn costs turns) + +|vex-context-firewall |EFR, ICS +|↑ TII -- retraction propagation costs turns + +|vex-scope-governor |SFR, PQ +|↑ SFR -- *same metric, opposite direction*: over-tight scope converts +creep into collapse, which METRICS.adoc scores as Critical + +|vex-error-recovery |RCI +|↑ TII -- strategy variation costs attempts + +|vex-verbosity-compressor |LPS, TII +|↑ CII -- compression removes content the user needed + +|vex-clarification-gate |PQ, TII +|↑ TII -- *same metric, opposite direction*: a gate is itself an +interruption +|=== + +The three same-metric entries are the sharpest cases. SFR and TII are +two-sided metrics: they penalise both excess and deficit. A satellite +targeting one side must demonstrate it has not simply crossed to the +other. + +== Protocol: vexometer-efficacy-v2 + +`vexometer-efficacy-v2` is *additive* to `vexometer-efficacy-v1` +(link:SATELLITES.adoc[SATELLITES.adoc]). v1 reports remain valid and +readable. v2 adds three fields: `collateral_metrics` (all ten), a +`capability` block, and a `verdict`. + +[source,json] +---- +{ + "version": "vexometer-efficacy-v2", + "satellite": "vex-verbosity-compressor", + "evaluation_date": "2026-09-01", + "sample_size": 500, + "scenario_set": "sha256:6b2f...", + "target_metrics": { + "LPS": { + "baseline": 0.41, + "after": 0.17, + "gap_closed": 0.585, + "mean_reduction": 0.24, + "std_dev": 0.09, + "confidence": 0.95, + "p_value": 0.001 + }, + "TII": { + "baseline": 0.33, + "after": 0.22, + "gap_closed": 0.333, + "mean_reduction": 0.11, + "std_dev": 0.07, + "confidence": 0.95, + "p_value": 0.004 + } + }, + "collateral_metrics": { + "EFR": { "baseline": 0.19, "after": 0.20, "delta": 0.01 }, + "PQ": { "baseline": 0.28, "after": 0.26, "delta": -0.02 }, + "TAI": { "baseline": 0.15, "after": 0.15, "delta": 0.00 }, + "ICS": { "baseline": 0.22, "after": 0.23, "delta": 0.01 }, + "CII": { "baseline": 0.31, "after": 0.35, "delta": 0.04 }, + "SRS": { "baseline": 0.26, "after": 0.26, "delta": 0.00 }, + "SFR": { "baseline": 0.24, "after": 0.25, "delta": 0.01 }, + "RCI": { "baseline": 0.30, "after": 0.30, "delta": 0.00 } + }, + "capability": { + "proxy": "data/probes/behavioural_probes.json", + "probes_total": 13, + "pass_rate_before": 0.923, + "pass_rate_after": 0.923, + "capability_ok": true + }, + "isa_delta": -2.61, + "verdict": "accept_with_warning", + "verdict_notes": "CII regressed by 0.04 -- compression removes content in long-form code scenarios. Must be declared in satellite README.", + "methodology": "A/B testing with vexometer validation", + "traces_available": true, + "frontier_record": "frontier/LPS-2026-09-01.json" +} +---- + +=== Migration from v1 + +[cols="2,3", options="header"] +|=== +|v1 field |v2 treatment + +|`metrics` |Renamed `target_metrics`; existing sub-fields preserved, +`baseline` / `after` / `gap_closed` added +|`sample_size`, `methodology`, `traces_available` |Unchanged +|-- |`collateral_metrics`, `capability`, `isa_delta`, `verdict`, +`scenario_set`, `frontier_record` added +|=== + +A v1 report can be mechanically lifted to v2 shape, but it cannot be +assigned a verdict: the collateral and capability evidence does not +exist. Lifted reports carry `"verdict": "unverified"`. + +== Protocol: vexometer-frontier-v1 + +An efficacy report states the outcome. A frontier record states the +*search*. It is an ordered log of every configuration tried against one +metric, whether accepted or rejected, so that a claim of "closes 58% of +the LPS gap" can be read alongside how many attempts it took and what was +discarded on the way. + +[source,json] +---- +{ + "version": "vexometer-frontier-v1", + "metric": "LPS", + "model_profile": "claude-opus-5", + "timestamp": "2026-09-01T10:30:00Z", + "scenario_set": "sha256:6b2f...", + "baseline": { + "LPS": 0.41, + "isa_score": 46.2, + "probe_pass_rate": 0.923 + }, + "attempts": [ + { + "index": 1, + "satellite": "vex-verbosity-compressor", + "config": "strip_filler=true,semantic_dedup=false", + "target_after": 0.34, + "gap_closed": 0.171, + "collateral_max": { "metric": "CII", "delta": 0.01 }, + "isa_delta": -0.94, + "capability_ok": true, + "verdict": "accept", + "frontier": 0.171 + }, + { + "index": 2, + "satellite": "vex-verbosity-compressor", + "config": "strip_filler=true,semantic_dedup=true,aggressive=true", + "target_after": 0.09, + "gap_closed": 0.780, + "collateral_max": { "metric": "CII", "delta": 0.11 }, + "isa_delta": 0.42, + "capability_ok": false, + "verdict": "reject_capability", + "frontier": 0.171 + }, + { + "index": 3, + "satellite": "vex-verbosity-compressor", + "config": "strip_filler=true,semantic_dedup=true,preserve_code=true", + "target_after": 0.17, + "gap_closed": 0.585, + "collateral_max": { "metric": "CII", "delta": 0.04 }, + "isa_delta": -2.61, + "capability_ok": true, + "verdict": "accept_with_warning", + "frontier": 0.585 + } + ], + "frontier_final": 0.585, + "methods_tried": 3, + "methods_rejected": 1 +} +---- + +*Invariant*: `frontier` is monotonically non-decreasing. It updates only +on an `accept` or `accept_with_warning` verdict whose `gap_closed` +exceeds the current frontier. A rejected attempt with a higher +`gap_closed` than the frontier does *not* advance it -- that is the whole +point of recording rejections. + +== The Frontier Plot + +The canonical rendering of a frontier record, and the intended shape of +the ISA efficacy dashboard: ten small multiples, one per metric. + +[cols="1,4", options="header"] +|=== +|Element |Meaning + +|X axis |`index` -- configurations tried, in order +|Y axis |`gap_closed` (%) for the metric +|Step line |`frontier` -- best accepted result so far +|Point |Each attempt with an `accept*` verdict +|Cross |Each attempt with a `reject*` verdict, annotated by reason +|Star |`frontier_final` +|=== + +The visual signature to read from a completed panel: + +* *Early plateau, few crosses* -- the metric is mechanically tractable; + further search is low-yield. +* *Late step change* -- a qualitatively different intervention was needed; + earlier configurations were exploring the wrong family. +* *Dense crosses above the frontier* -- the metric is entangled with + capability or with a neighbouring metric. Interventions that "work" on + the target keep failing acceptance. This is the signal that a metric + needs a *different satellite*, not a better-tuned one. + +== Search Budget Tiers + +link:SATELLITES.adoc[SATELLITES.adoc] prioritises satellites by *impact*. +This protocol adds a second, orthogonal axis: *expected search depth* -- +how many configurations a metric is likely to need before its frontier +flattens. + +The following tiers are a *hypothesis to be tested by the frontier +records*, not a measured result. They should be revised once real +frontier data exists. + +[cols="1,2,4", options="header"] +|=== +|Tier |Metrics |Rationale + +|Fast (target ~50 attempts) +|CII, SFR, TII +|Mechanically detectable. Success criteria are crisp (no placeholders; no +scope deviation; no unsolicited output). Detection is close to a parsing +problem. + +|Medium (target ~120 attempts) +|ICS, SRS, RCI, PQ +|Detectable from interaction structure -- repeats, loops, patch chains -- +but the threshold between "persistent" and "rigid" is judgement-laden. + +|Slow (target ~200+ attempts) +|LPS, EFR, TAI +|Entangled with capability and with each other. Suppressing sycophancy +risks paternalism; calibrating confidence raises hedge density; TAI is +substantially a policy surface rather than a behavioural one. +|=== + +*Planning implication*: a satellite in the High Priority tier that +targets a Slow metric (vex-sycophancy-shield, vex-hallucination-guard) +should be scoped with a *lower* expected frontier and a *larger* attempt +budget than its priority alone suggests. An early efficacy claim of a +large reduction on a Slow metric warrants more scrutiny, not less. + +== Workflow + +Extends the five steps in link:SATELLITES.adoc[SATELLITES.adoc] +("Validating Satellite Efficacy") with collateral and capability +measurement. + +. *Fix the scenario set.* Content-address it. Every attempt in a frontier + record must run against the identical set, or the frontier is + meaningless. +. *Collect baseline.* All ten metrics plus probe pass-rate -- not only the + target metric. +. *Apply the satellite configuration.* +. *Collect post-intervention.* All ten metrics plus probe pass-rate. +. *Compute* `G_m`, all `D_c`, `D_ISA`, `capability_ok`. +. *Assign a verdict* from the acceptance rule table. +. *Append to the frontier record*, whether accepted or rejected. +. *On acceptance*, emit a `vexometer-efficacy-v2` report referencing the + frontier record. + +=== Proposed Tooling + +These recipes do not yet exist in the Justfile. They are specified here +alongside the aspirational `trace-collect` / `efficacy-report` recipes +already described in link:SATELLITES.adoc[SATELLITES.adoc]. + +[source,bash] +---- +# Baseline all ten metrics plus the probe suite +just efficacy-baseline --scenarios SET --output baseline.json + +# Evaluate one configuration; appends to the frontier record +just efficacy-attempt --satellite NAME --config CFG \ + --baseline baseline.json --frontier frontier/METRIC.json + +# Emit a vexometer-efficacy-v2 report from an accepted attempt +just efficacy-report --frontier frontier/METRIC.json --output EFFICACY.adoc + +# Render the ten-panel frontier plot +just efficacy-plot --frontier-dir frontier/ --output docs/frontier.svg +---- + +=== Satellite Repository Additions + +The satellite repository structure in +link:SATELLITES.adoc[SATELLITES.adoc] gains one directory: + +[source] +---- +vexometer/ +├── before/ +├── after/ +├── frontier/ # NEW: vexometer-frontier-v1 records +│ └── METRIC-DATE.json # one per metric searched +└── EFFICACY.adoc +---- + +A satellite claiming efficacy without a frontier record is making an +unaudited claim. ISA should report such claims as `unverified`. + +== Scope Boundary + +This protocol governs the *irritation surface* only. It is not a safety +or alignment evaluation and must not be presented as one. + +ISA's ten metrics measure what makes a tool exhausting to work with: +time wasted, tics, incompleteness, rigidity, scope drift, poor recovery. +Several of these -- paternalism, strategic rigidity, recovery competence +-- have no counterpart in safety evaluation, because they endanger nobody. +Conversely, ISA measures nothing about deception, power-seeking, or +jailbreak resistance, and a satellite passing every check in this +document implies nothing about those properties. + +The overlap that does exist is narrow and worth stating precisely: + +[cols="2,2,3", options="header"] +|=== +|Safety-evaluation concern |Nearest ISA metric |Relationship + +|Sycophancy |LPS |Substantial overlap +|Hallucination |EFR |Substantial overlap +|Concealing uncertainty |EFR (calibration term) |Partial -- ISA scores it +as miscalibration, not concealment +|Privacy violation |TAI |Partial -- see note below +|Reward hacking |CII, SFR |Weak analogy -- satisfying the letter of a +request while gutting its substance is scope mutation plus stub returns +|=== + +NOTE: TAI is currently defined as a policy surface with no planned +satellites. That framing conflates two separable things: product +telemetry policy, which is genuinely not addressable in-band, and model +behaviour when handling sensitive material in context, which is. Splitting +TAI would give it a tractable satellite target. Recorded here as an open +question against link:METRICS.adoc[METRICS.adoc]. + +== Appendix: Provenance and Rationale + +_Non-normative. The protocol above stands independently of this section._ + +The frontier-record design was informed by a published Anthropic figure +titled "Iterative improvement against ten alignment failures", which +plots safety gap closed (%) against methods tried, in order, across ten +failure modes, with a best-method-so-far step function and explicit +markers for methods *rejected because they hurt capability*. Provenance +has not been independently verified beyond the figure itself; it is cited +here as design inspiration, not as evidence, and deliberately has no +entry in link:CITATIONS.adoc[CITATIONS.adoc]. + +Two ideas were taken from it. + +*The rejection marker.* The figure treats "this method closed the gap but +cost capability" as a first-class, recorded outcome rather than a +discarded experiment. `vexometer-efficacy-v1` had no way to express this; +the acceptance rule and the `reject_*` verdicts exist to close that gap. + +*The ordered search.* Plotting attempts in sequence rather than reporting +a final number makes the shape of the search legible -- fast convergence, +late breakthrough, or persistent entanglement. + +The figure's reported outcomes, offered only as weak external support for +the Search Budget Tiers hypothesis: the two dimensions closest to ISA's +LPS and EFR resisted intervention hardest (sycophancy ~26% of gap closed +after 200+ methods; hallucination ~40%; concealing uncertainty ~39%), +while the most mechanically specifiable dimension converged fastest +(reward hacking ~96% in ~40 methods). Attempt budgets varied by roughly +5x across dimensions. + +This is suggestive, not transferable. The measurements are against a +different baseline, on a different construct, by a different method. It +is offered as a reason to *expect* uneven search depth across ISA's ten +metrics -- and to build the instrument that measures it -- not as a +prediction of what ISA will find. + +== See Also + +* link:METRICS.adoc[Metrics Reference] - All 10 ISA metrics and category weights +* link:SATELLITES.adoc[Satellite Architecture] - Intervention tools, trace and + efficacy-v1 formats +* link:SPECIFICATION.adoc[Full Specification] - Technical specification document diff --git a/vexometer/docs/METRICS.adoc b/vexometer/docs/METRICS.adoc index 3729f78..f5b0bfb 100644 --- a/vexometer/docs/METRICS.adoc +++ b/vexometer/docs/METRICS.adoc @@ -487,4 +487,5 @@ ISA = sum(category_score * category_weight) / sum(category_weight) * 100 == See Also * link:SATELLITES.adoc[Satellite Architecture] - Intervention tools that reduce metrics +* link:EFFICACY-PROTOCOL.adoc[Efficacy Protocol] - Cross-metric acceptance rule using the category weights above * link:SPECIFICATION.md[Full Specification] - Technical specification document diff --git a/vexometer/docs/SATELLITES.adoc b/vexometer/docs/SATELLITES.adoc index 8bf50af..c183d22 100644 --- a/vexometer/docs/SATELLITES.adoc +++ b/vexometer/docs/SATELLITES.adoc @@ -414,5 +414,6 @@ All satellites must follow: == See Also * link:METRICS.adoc[Metrics Reference] - All 10 ISA metrics (vexometer identifiers) +* link:EFFICACY-PROTOCOL.adoc[Efficacy Protocol] - Acceptance rule and frontier records extending vexometer-efficacy-v1 * link:../ECOSYSTEM.scm[Ecosystem Map] - Satellite relationships * link:../META.scm[Project Metadata] - ISA meta-information From f2c749d3e4e06bc04b07222116d941efd8847d27 Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" <6759885+hyperpolymath@users.noreply.github.com> Date: Tue, 1 Sep 2026 03:59:41 +0100 Subject: [PATCH 2/6] fix(vexometer): correct worked example and define verdict precedence Two defects in EFFICACY-PROTOCOL.adoc's own worked examples, both on the surface the protocol claims to make auditable: - isa_delta was stated as -2.61 in both the efficacy-v2 report and the matching frontier-v1 attempt. Recomputing from the example's own deltas against the METRICS.adoc category weights (sum 11.9) gives a weighted delta of -0.323 and so D_ISA = -2.71. A specification whose only fully worked example fails its own acceptance formula is the first thing an implementer trips on. - Verdict precedence was undefined while the example depended on it. Frontier attempt 2 satisfies both reject_collateral (CII +0.11) and reject_capability, and was labelled reject_capability with nothing in the document justifying that choice. Adds an explicit ordering, reject_null > reject_capability > reject_collateral > reject_net, noted as deliberately not the table order. Also flags that the proposed efficacy-report signature supersedes the --before/--after sketch in SATELLITES.adoc, since neither is implemented. Co-Authored-By: Claude Opus 5 --- vexometer/docs/EFFICACY-PROTOCOL.adoc | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/vexometer/docs/EFFICACY-PROTOCOL.adoc b/vexometer/docs/EFFICACY-PROTOCOL.adoc index 935fe9d..877395a 100644 --- a/vexometer/docs/EFFICACY-PROTOCOL.adoc +++ b/vexometer/docs/EFFICACY-PROTOCOL.adoc @@ -128,6 +128,18 @@ satellite README. |Does not advance |=== +*Verdict precedence*: more than one reject condition can hold for a single +attempt. When they do, report the most fundamental cause, in this order: + +[source] +---- +reject_null > reject_capability > reject_collateral > reject_net +---- + +A verdict is therefore always a single value, and two implementations of +this protocol assign the same verdict to the same data. Note that this is +*not* the order the conditions appear in the table above. + *Provisional constants*: the `0.02` / `0.05` collateral thresholds and the one-probe capability tolerance are stated so that evaluations are reproducible today. They are expected to be recalibrated once enough @@ -242,7 +254,7 @@ readable. v2 adds three fields: `collateral_metrics` (all ten), a "pass_rate_after": 0.923, "capability_ok": true }, - "isa_delta": -2.61, + "isa_delta": -2.71, "verdict": "accept_with_warning", "verdict_notes": "CII regressed by 0.04 -- compression removes content in long-form code scenarios. Must be declared in satellite README.", "methodology": "A/B testing with vexometer validation", @@ -321,7 +333,7 @@ discarded on the way. "target_after": 0.17, "gap_closed": 0.585, "collateral_max": { "metric": "CII", "delta": 0.04 }, - "isa_delta": -2.61, + "isa_delta": -2.71, "capability_ok": true, "verdict": "accept_with_warning", "frontier": 0.585 @@ -441,6 +453,8 @@ just efficacy-attempt --satellite NAME --config CFG \ --baseline baseline.json --frontier frontier/METRIC.json # Emit a vexometer-efficacy-v2 report from an accepted attempt +# (supersedes the --before/--after signature sketched in SATELLITES.adoc; +# neither is implemented yet) just efficacy-report --frontier frontier/METRIC.json --output EFFICACY.adoc # Render the ten-panel frontier plot From 2e47e56fffdf4400b8616a87752af96e2b0bd290 Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" <6759885+hyperpolymath@users.noreply.github.com> Date: Tue, 1 Sep 2026 04:07:28 +0100 Subject: [PATCH 3/6] docs(vexometer): fix broken cross-links and correct data-file counts Follow-up to the efficacy protocol. Verified every relative link: target in vexometer/ resolves, and every count claimed for a data file against the file itself. Broken links (the AsciiDoc migration left these pointing at .md files that no longer exist): - docs/SPECIFICATION.md -> .adoc (README, ROADMAP x2, METRICS) - docs/letter_lmsys_arena.md -> .adoc (README, ROADMAP) - ../NAMING.md -> ../NAMING.adoc (README) - LICENSE.txt -> LICENSE (README; the file has no extension) Also corrected in the README directory tree. Data-file counts in the ROADMAP table, two of three were wrong: - behavioural_probes.json: 14 -> 13. The suite is designed as seven categories of two probes, but PROBE-CORRECTION-002 is absent, so the shipped file holds 13. Recorded as "13 of 14" with a note, rather than silently restated as 13, because the gap looks unintended. Probe_Count is computed from the loaded suite, so no code depends on the literal. - linguistic_pathology.json: 16 -> 15. Plain off-by-one; the ids show no structural gap (6 sycophancy, 2 identity, 4 hedge, 3 corporate). - paternalism.json: 12, verified correct. Consequently the capability tolerance in EFFICACY-PROTOCOL.adoc is now expressed as 1/N over the loaded suite rather than the literal 1/13, with a note to re-baseline when the missing probe lands. Adds EFFICACY-PROTOCOL.adoc to the ROADMAP documentation index and See Also, and notes in SATELLITES.adoc that its --before/--after efficacy-report sketch is superseded by the --frontier signature. Two dangling links remain and are left alone deliberately: docs/SATELLITES.adoc references ../ECOSYSTEM.scm and ../META.scm, neither of which exists anywhere in the repository. Creating or removing them is a design decision, not a link fix. Co-Authored-By: Claude Opus 5 --- vexometer/README.adoc | 12 ++++++------ vexometer/ROADMAP.adoc | 22 ++++++++++++++++------ vexometer/docs/EFFICACY-PROTOCOL.adoc | 9 ++++++--- vexometer/docs/METRICS.adoc | 2 +- vexometer/docs/SATELLITES.adoc | 2 ++ 5 files changed, 31 insertions(+), 16 deletions(-) diff --git a/vexometer/README.adoc b/vexometer/README.adoc index b68f79b..52356f6 100644 --- a/vexometer/README.adoc +++ b/vexometer/README.adoc @@ -20,7 +20,7 @@ A rigorous, reproducible tool for quantifying the irritation surface of LLMs and *The Vexometer: Irritation Surface Analyser (ISA) for LLMs and related tools* is the canonical title. *ISA* is the canonical short name. *Vexometer* is retained as a compatibility codename in technical identifiers such as `vexometer/`, `vexometer.gpr`, the `vexometer` CLI command, and `Vexometer.*` Ada packages. -The repository-wide naming policy is defined in link:../NAMING.md[NAMING.md]. +The repository-wide naming policy is defined in link:../NAMING.adoc[NAMING.adoc]. == Philosophy @@ -193,11 +193,11 @@ vexometer/ | | +-- behavioural_probes.json | +-- baselines/ # Known model baselines +-- docs/ -| +-- SPECIFICATION.md # Full technical specification +| +-- SPECIFICATION.adoc # Full technical specification | +-- METRICS.adoc # All 10 metrics detailed | +-- SATELLITES.adoc # Intervention satellite architecture | +-- EFFICACY-PROTOCOL.adoc # Satellite acceptance and frontier records -| +-- letter_lmsys_arena.md # LMSYS Arena proposal +| +-- letter_lmsys_arena.adoc # LMSYS Arena proposal +-- alire.toml # Alire package manifest +-- vexometer.gpr # GNAT project file ---- @@ -350,7 +350,7 @@ See link:docs/SATELLITES.adoc[SATELLITES.adoc] for the full satellite architectu == LMSYS Arena Integration -ISA includes a proposal for integrating ISA metrics into the LMSYS Chatbot Arena evaluation framework. See link:docs/letter_lmsys_arena.md[letter_lmsys_arena.md]. +ISA includes a proposal for integrating ISA metrics into the LMSYS Chatbot Arena evaluation framework. See link:docs/letter_lmsys_arena.adoc[letter_lmsys_arena.adoc]. Preliminary testing shows significant variation in irritation surfaces across models: @@ -403,7 +403,7 @@ Priority areas: == Documentation -* link:docs/SPECIFICATION.md[SPECIFICATION.md] - Full technical specification +* link:docs/SPECIFICATION.adoc[SPECIFICATION.adoc] - Full technical specification * link:docs/METRICS.adoc[METRICS.adoc] - Detailed metric reference * link:docs/SATELLITES.adoc[SATELLITES.adoc] - Satellite architecture * link:docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc] - How satellite efficacy claims are accepted or rejected @@ -413,6 +413,6 @@ Priority areas: image:https://img.shields.io/badge/License-PMPL--1.0--or--later-6366f1?style=flat&labelColor=4338ca[MPL-2.0,link="https://github.com/hyperpolymath/palimpsest-license"] -This project is licensed under MPL-2.0 (Palimpsest License). See link:LICENSE.txt[LICENSE.txt] for the full license text. +This project is licensed under MPL-2.0 (Palimpsest License). See link:LICENSE[LICENSE] for the full license text. This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. diff --git a/vexometer/ROADMAP.adoc b/vexometer/ROADMAP.adoc index 87058cb..fc815cc 100644 --- a/vexometer/ROADMAP.adoc +++ b/vexometer/ROADMAP.adoc @@ -29,7 +29,7 @@ Development roadmap for ISA (Irritation Surface Analyser, formerly Vexometer). |Implemented (built-ins) |*Probe System* -|Behavioural probe framework, 14 standardised probes, probe suite runner +|Behavioural probe framework, 13 of 14 standardised probes, probe suite runner |Implemented (built-ins) |*API Clients* @@ -59,17 +59,26 @@ Development roadmap for ISA (Irritation Surface Analyser, formerly Vexometer). |=== |Data File |Items |Categories -|`linguistic_pathology.json` |16 patterns |Sycophancy, identity, hedge, corporate +|`linguistic_pathology.json` |15 patterns |Sycophancy, identity, hedge, corporate |`paternalism.json` |12 patterns |Warning, lecture, competence, refusal -|`behavioural_probes.json` |14 probes |Brevity, competence, sycophancy, constraint, uncertainty, direct +|`behavioural_probes.json` |13 probes |Brevity, competence, sycophancy, correction, constraint, uncertainty, direct |=== +NOTE: The probe suite is designed as seven categories of two probes (14). +`PROBE-CORRECTION-002` is not yet present, so the shipped suite holds 13. +`Probe_Count` is computed from the loaded suite, so nothing depends on the +literal count; but the capability proxy in +link:docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc] is expressed as a +pass-rate over the suite, and adding the missing probe will shift baselines +recorded before it lands. + === Documentation -* link:docs/SPECIFICATION.md[SPECIFICATION.md] - Full technical specification +* link:docs/SPECIFICATION.adoc[SPECIFICATION.adoc] - Full technical specification * link:docs/METRICS.adoc[METRICS.adoc] - All 10 metrics with calculation details * link:docs/SATELLITES.adoc[SATELLITES.adoc] - Satellite architecture and integration protocol -* link:docs/letter_lmsys_arena.md[letter_lmsys_arena.md] - LMSYS Arena proposal letter +* link:docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc] - Satellite acceptance rule and frontier records +* link:docs/letter_lmsys_arena.adoc[letter_lmsys_arena.adoc] - LMSYS Arena proposal letter == Phase 1: Core Implementation @@ -383,6 +392,7 @@ Contributions welcome in all areas. See link:CONTRIBUTING.adoc[CONTRIBUTING.adoc == See Also * link:README.adoc[README.adoc] - Project overview -* link:docs/SPECIFICATION.md[SPECIFICATION.md] - Technical specification +* link:docs/SPECIFICATION.adoc[SPECIFICATION.adoc] - Technical specification * link:docs/METRICS.adoc[METRICS.adoc] - Metric definitions * link:docs/SATELLITES.adoc[SATELLITES.adoc] - Satellite architecture +* link:docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc] - Efficacy acceptance rule diff --git a/vexometer/docs/EFFICACY-PROTOCOL.adoc b/vexometer/docs/EFFICACY-PROTOCOL.adoc index 877395a..f35c3d8 100644 --- a/vexometer/docs/EFFICACY-PROTOCOL.adoc +++ b/vexometer/docs/EFFICACY-PROTOCOL.adoc @@ -83,9 +83,12 @@ on baseline output and on satellite-processed output. |`capability_ok` |`P_after >= P_before - tolerance` |=== -*Provisional tolerance*: one probe (`1/13 ~= 0.077`). A satellite that -fails two or more probes its baseline passed is rejected regardless of -metric improvement. +*Provisional tolerance*: one probe, i.e. `1/N` for a suite of `N` probes. +A satellite that fails two or more probes its baseline passed is rejected +regardless of metric improvement. The shipped suite currently holds 13 of +its designed 14 probes (`PROBE-CORRECTION-002` is absent), so the tolerance +is presently `1/13 ~= 0.077`. Express it against the loaded suite size +rather than a literal, and re-baseline when the missing probe lands. NOTE: The probe suite is a proxy, not a capability benchmark. It detects gross degradation, not subtle loss. Satellites whose interventions are diff --git a/vexometer/docs/METRICS.adoc b/vexometer/docs/METRICS.adoc index f5b0bfb..d938392 100644 --- a/vexometer/docs/METRICS.adoc +++ b/vexometer/docs/METRICS.adoc @@ -488,4 +488,4 @@ ISA = sum(category_score * category_weight) / sum(category_weight) * 100 * link:SATELLITES.adoc[Satellite Architecture] - Intervention tools that reduce metrics * link:EFFICACY-PROTOCOL.adoc[Efficacy Protocol] - Cross-metric acceptance rule using the category weights above -* link:SPECIFICATION.md[Full Specification] - Technical specification document +* link:SPECIFICATION.adoc[Full Specification] - Technical specification document diff --git a/vexometer/docs/SATELLITES.adoc b/vexometer/docs/SATELLITES.adoc index c183d22..5baa082 100644 --- a/vexometer/docs/SATELLITES.adoc +++ b/vexometer/docs/SATELLITES.adoc @@ -332,6 +332,8 @@ Compare before and after metrics: [source,bash] ---- +# Superseded by the --frontier signature in EFFICACY-PROTOCOL.adoc, +# which records collateral metrics and capability. Neither is implemented yet. just efficacy-report --before before/ --after after/ --output EFFICACY.adoc ---- From 89a4c42dbdd65bef2867055375c8fd8e485934e7 Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" <6759885+hyperpolymath@users.noreply.github.com> Date: Tue, 1 Sep 2026 13:51:18 +0000 Subject: [PATCH 4/6] docs(efficacy): verify provenance appendix against the published paper The appendix previously described its source figure from an unattributed reproduction and was marked provenance-unverified. Checked against the primary source (Anthropic, 'Automated Researchers Can Reliably Mitigate Alignment Failures', 2026-08-28): corrected the figure identification to Fig. 3 and its real caption, cited the announcement and paper URLs, stated the three rejection rules incl. the 95%-CI capability gate, confirmed the per-dimension numbers from the figure's own panel labels (n = 44..221 ~= the 5x budget spread the tiers assume), and added the Section-6 result as motivation for machine-readable frontier records. Co-Authored-By: Claude Fable 5 --- vexometer/docs/EFFICACY-PROTOCOL.adoc | 64 ++++++++++++++++++++------- 1 file changed, 49 insertions(+), 15 deletions(-) diff --git a/vexometer/docs/EFFICACY-PROTOCOL.adoc b/vexometer/docs/EFFICACY-PROTOCOL.adoc index f35c3d8..8b3646c 100644 --- a/vexometer/docs/EFFICACY-PROTOCOL.adoc +++ b/vexometer/docs/EFFICACY-PROTOCOL.adoc @@ -521,14 +521,26 @@ question against link:METRICS.adoc[METRICS.adoc]. _Non-normative. The protocol above stands independently of this section._ -The frontier-record design was informed by a published Anthropic figure -titled "Iterative improvement against ten alignment failures", which -plots safety gap closed (%) against methods tried, in order, across ten -failure modes, with a best-method-so-far step function and explicit -markers for methods *rejected because they hurt capability*. Provenance -has not been independently verified beyond the figure itself; it is cited -here as design inspiration, not as evidence, and deliberately has no -entry in link:CITATIONS.adoc[CITATIONS.adoc]. +The frontier-record design was informed by Figure 3 of an Anthropic +paper, since verified against the primary source: + +* Chen Yueh-Han, Jiaxin Wen, and Jan Hendrik Kirchner, _Automated + Researchers Can Reliably Mitigate Alignment Failures_. Anthropic + Fellows Program / Anthropic / UC Berkeley, announced 2026-08-28. +* Announcement: https://alignment.anthropic.com/2026/automated-alignment-researchers/ +* Paper: https://www-cdn.anthropic.com/7b1c44894e980876479947dcdd40716278aeeffd/automated-alignment-researchers-august-2026.pdf + +(link:CITATIONS.adoc[CITATIONS.adoc] is this project's how-to-cite-ISA +guide, not a reference list, so the reference lives here.) + +Figure 3, "AARs reliably hill-climb every alignment failure", plots for +each of ten alignment failures the geometric-mean *headroom closed* (%) +on that failure's benchmarks, for every proposed method in chronological +order, with a best-valid-method-so-far step line, a star on the winning +method, and crosses for methods *filtered out for degrading a capability +benchmark*. An earlier draft of this appendix described the figure from +an unattributed reproduction; the description and the numbers below have +since been checked against the paper itself. Two ideas were taken from it. @@ -541,13 +553,27 @@ the acceptance rule and the `reject_*` verdicts exist to close that gap. a final number makes the shape of the search legible -- fast convergence, late breakthrough, or persistent entanglement. -The figure's reported outcomes, offered only as weak external support for -the Search Budget Tiers hypothesis: the two dimensions closest to ISA's -LPS and EFR resisted intervention hardest (sycophancy ~26% of gap closed -after 200+ methods; hallucination ~40%; concealing uncertainty ~39%), -while the most mechanically specifiable dimension converged fastest -(reward hacking ~96% in ~40 methods). Attempt budgets varied by roughly -5x across dimensions. +The source's rejection rules are worth stating precisely, since this +protocol's machinery is their analogue. A method is disqualified, +whatever its safety score, if the trained model's 95% confidence +interval on any capability benchmark falls entirely below the base +model's; training on benchmark or evaluation data is forbidden; and so +is distilling from stronger models. The first maps to `capability_ok`. +The second's analogue is that a satellite must never be tuned against +the scenario set it is scored on -- the content-addressed `scenario_set` +makes that auditable, though a held-out scenario set does not yet exist +and is future work. The third has no satellite analogue yet; it is noted +here for when satellites start delegating to models. + +The figure's reported outcomes (its own panel labels; n = methods +proposed), offered as weak external support for the Search Budget Tiers +hypothesis: the two dimensions closest to ISA's LPS and EFR resisted +intervention hardest -- sycophancy closed 26% of headroom in 221 +methods, hallucination 40% in 200, concealing uncertainty 39% in 188 -- +while the most mechanically specifiable dimension converged fastest: +reward hacking closed 96% in 44 methods. Attempt budgets spanned n = 44 +to n = 221 across the ten failures, almost exactly the ~5x spread the +tiers assume. This is suggestive, not transferable. The measurements are against a different baseline, on a different construct, by a different method. It @@ -555,6 +581,14 @@ is offered as a reason to *expect* uneven search depth across ISA's ten metrics -- and to build the instrument that measures it -- not as a prediction of what ISA will find. +One further result motivates keeping frontier records machine-readable. +In the paper's Section 6, an automated researcher (Claude Sonnet 5) +post-trained an early Claude Opus 4.8 checkpoint to near its released +alignment scores with about 2,400 training examples. The plausible +future consumer of a frontier record is another automated searcher, not +only a human reader -- an argument for strict formats now, while the +records are still cheap to standardise. + == See Also * link:METRICS.adoc[Metrics Reference] - All 10 ISA metrics and category weights From 987dde44bb6faede46947ace234a77ef85925237 Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" <6759885+hyperpolymath@users.noreply.github.com> Date: Tue, 1 Sep 2026 15:26:15 +0100 Subject: [PATCH 5/6] docs(efficacy): resolve six review consistency findings - gap_closed: formula said percentage, every example stores a fraction; formula now says fraction in [0, 1] and the JSON contract is stated - D_ISA: sum domain made explicit (all ten metrics, targets included) - same-metric risk rows: the table has two, prose said three - collateral_metrics: holds non-target metrics, prose said all ten - unverified: defined as a verification-status sentinel outside the six-verdict acceptance table and its precedence - SATELLITES.adoc: superseded just efficacy-report promoted from an in-block comment to a NOTE; command kept as documented shape only Normative findings (zero-baseline G_m, per-probe capability identity, multi-target rule, frontier plurality, v1 to v2 migration mechanics, held-out scenario set) are deferred to the sitrep route-forward: the protocol is one day old and this pass makes no acceptance-rule changes. Co-Authored-By: Claude Fable 5 --- vexometer/docs/EFFICACY-PROTOCOL.adoc | 19 ++++++++++++++----- vexometer/docs/SATELLITES.adoc | 8 ++++++-- 2 files changed, 20 insertions(+), 7 deletions(-) diff --git a/vexometer/docs/EFFICACY-PROTOCOL.adoc b/vexometer/docs/EFFICACY-PROTOCOL.adoc index 8b3646c..3b690c3 100644 --- a/vexometer/docs/EFFICACY-PROTOCOL.adoc +++ b/vexometer/docs/EFFICACY-PROTOCOL.adoc @@ -43,9 +43,12 @@ The fraction of the available irritation gap removed. Since the floor is [source] ---- -G_m = (B_m - A_m) / B_m expressed as a percentage +G_m = (B_m - A_m) / B_m a fraction in [0, 1] ---- +Reports store `gap_closed` as this fraction (`0.585`, not `58.5`); +prose may cite it as a percentage. + *Collateral delta* (`D_c`):: For every metric `c` other than the target(s), the absolute change `A_c - B_c`. Positive means the metric got worse. @@ -59,6 +62,9 @@ in link:METRICS.adoc[METRICS.adoc]: D_ISA = ( SUM(w_c * (A_c - B_c)) / SUM(w_c) ) * 100 ---- +Here the sum ranges over all ten metrics -- targets included -- unlike +the collateral set above, which excludes them. + A satellite must produce `D_ISA < 0` to be accepted. This is the rule that prevents a large reduction in a low-weight metric from paying for a small regression in a high-weight one. @@ -200,7 +206,7 @@ creep into collapse, which METRICS.adoc scores as Critical interruption |=== -The three same-metric entries are the sharpest cases. SFR and TII are +The two same-metric entries are the sharpest cases. SFR and TII are two-sided metrics: they penalise both excess and deficit. A satellite targeting one side must demonstrate it has not simply crossed to the other. @@ -209,8 +215,8 @@ other. `vexometer-efficacy-v2` is *additive* to `vexometer-efficacy-v1` (link:SATELLITES.adoc[SATELLITES.adoc]). v1 reports remain valid and -readable. v2 adds three fields: `collateral_metrics` (all ten), a -`capability` block, and a `verdict`. +readable. v2 adds three fields: `collateral_metrics` (every metric outside +`target_metrics`), a `capability` block, and a `verdict`. [source,json] ---- @@ -281,7 +287,10 @@ readable. v2 adds three fields: `collateral_metrics` (all ten), a A v1 report can be mechanically lifted to v2 shape, but it cannot be assigned a verdict: the collateral and capability evidence does not -exist. Lifted reports carry `"verdict": "unverified"`. +exist. Lifted reports carry `"verdict": "unverified"`. `unverified` is a +verification-status sentinel, not a seventh verdict: it sits outside the +acceptance-rule table and its precedence order, and marks a report whose +collateral and capability evidence is absent. == Protocol: vexometer-frontier-v1 diff --git a/vexometer/docs/SATELLITES.adoc b/vexometer/docs/SATELLITES.adoc index 5baa082..e33d3f1 100644 --- a/vexometer/docs/SATELLITES.adoc +++ b/vexometer/docs/SATELLITES.adoc @@ -330,10 +330,14 @@ just trace-collect --input responses/ --output after/ Compare before and after metrics: +NOTE: `just efficacy-report` is not yet implemented, and this v1 +signature is superseded by the `--frontier` workflow in +link:EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc], which records +collateral metrics and capability. The command below documents the +intended v1 shape only. + [source,bash] ---- -# Superseded by the --frontier signature in EFFICACY-PROTOCOL.adoc, -# which records collateral metrics and capability. Neither is implemented yet. just efficacy-report --before before/ --after after/ --output EFFICACY.adoc ---- From 31ac69a321b83da0beee209c4a95f9de1399a644 Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" <6759885+hyperpolymath@users.noreply.github.com> Date: Tue, 1 Sep 2026 15:26:59 +0100 Subject: [PATCH 6/6] chore(trust): regenerate vexometer manifest for the doc changes README.adoc and ROADMAP.adoc are manifest-tracked; any PR touching them must regenerate in the same PR or must-gates reds on the merge ref. Co-Authored-By: Claude Fable 5 --- vexometer/.trust/trust-manifest.sha256 | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/vexometer/.trust/trust-manifest.sha256 b/vexometer/.trust/trust-manifest.sha256 index d52aeee..4d1bd2e 100644 --- a/vexometer/.trust/trust-manifest.sha256 +++ b/vexometer/.trust/trust-manifest.sha256 @@ -1,8 +1,8 @@ # trust-manifest v1 # component=vexometer -# generated_at=2026-09-01T14:22:52Z -33d01034c9291410bad3c4ad057d92b031a7601fbe58cb98c382efd51d65639e README.adoc -730ed75dec35cb49730d2fb2e472f3e7752644ce7ccadee6b377e6b6c8cdda70 ROADMAP.adoc +# generated_at=2026-09-01T14:26:59Z +a0a0bd752f3f42324813e305648ee4a248b664516b17ef89e05d51b302f7765f README.adoc +394cb48ee253a28d448761e0714f26af0a38cb8ec408968d1220419b1035b3e7 ROADMAP.adoc f4358bff9ed297a634d7cef1d5efcc059de238d3bf6a73abd11f9e7910785d61 SECURITY.adoc 29db6e0287244a2668bc3a143c8b81a54a9ae762b98cd9683ee4c7b203a1cb8e contractiles/must/Mustfile 5f5e879cc11595c1f4228c41f3bbde5177584316d60f695211b7812b8de0565e contractiles/trust/Trustfile.a2ml