diff --git a/vexometer/.trust/trust-manifest.sha256 b/vexometer/.trust/trust-manifest.sha256 index d52aeee..4d1bd2e 100644 --- a/vexometer/.trust/trust-manifest.sha256 +++ b/vexometer/.trust/trust-manifest.sha256 @@ -1,8 +1,8 @@ # trust-manifest v1 # component=vexometer -# generated_at=2026-09-01T14:22:52Z -33d01034c9291410bad3c4ad057d92b031a7601fbe58cb98c382efd51d65639e README.adoc -730ed75dec35cb49730d2fb2e472f3e7752644ce7ccadee6b377e6b6c8cdda70 ROADMAP.adoc +# generated_at=2026-09-01T14:26:59Z +a0a0bd752f3f42324813e305648ee4a248b664516b17ef89e05d51b302f7765f README.adoc +394cb48ee253a28d448761e0714f26af0a38cb8ec408968d1220419b1035b3e7 ROADMAP.adoc f4358bff9ed297a634d7cef1d5efcc059de238d3bf6a73abd11f9e7910785d61 SECURITY.adoc 29db6e0287244a2668bc3a143c8b81a54a9ae762b98cd9683ee4c7b203a1cb8e contractiles/must/Mustfile 5f5e879cc11595c1f4228c41f3bbde5177584316d60f695211b7812b8de0565e contractiles/trust/Trustfile.a2ml diff --git a/vexometer/README.adoc b/vexometer/README.adoc index ab96121..52356f6 100644 --- a/vexometer/README.adoc +++ b/vexometer/README.adoc @@ -20,7 +20,7 @@ A rigorous, reproducible tool for quantifying the irritation surface of LLMs and *The Vexometer: Irritation Surface Analyser (ISA) for LLMs and related tools* is the canonical title. *ISA* is the canonical short name. *Vexometer* is retained as a compatibility codename in technical identifiers such as `vexometer/`, `vexometer.gpr`, the `vexometer` CLI command, and `Vexometer.*` Ada packages. -The repository-wide naming policy is defined in link:../NAMING.md[NAMING.md]. +The repository-wide naming policy is defined in link:../NAMING.adoc[NAMING.adoc]. == Philosophy @@ -193,10 +193,11 @@ vexometer/ | | +-- behavioural_probes.json | +-- baselines/ # Known model baselines +-- docs/ -| +-- SPECIFICATION.md # Full technical specification +| +-- SPECIFICATION.adoc # Full technical specification | +-- METRICS.adoc # All 10 metrics detailed | +-- SATELLITES.adoc # Intervention satellite architecture -| +-- letter_lmsys_arena.md # LMSYS Arena proposal +| +-- EFFICACY-PROTOCOL.adoc # Satellite acceptance and frontier records +| +-- letter_lmsys_arena.adoc # LMSYS Arena proposal +-- alire.toml # Alire package manifest +-- vexometer.gpr # GNAT project file ---- @@ -349,7 +350,7 @@ See link:docs/SATELLITES.adoc[SATELLITES.adoc] for the full satellite architectu == LMSYS Arena Integration -ISA includes a proposal for integrating ISA metrics into the LMSYS Chatbot Arena evaluation framework. See link:docs/letter_lmsys_arena.md[letter_lmsys_arena.md]. +ISA includes a proposal for integrating ISA metrics into the LMSYS Chatbot Arena evaluation framework. See link:docs/letter_lmsys_arena.adoc[letter_lmsys_arena.adoc]. Preliminary testing shows significant variation in irritation surfaces across models: @@ -402,15 +403,16 @@ Priority areas: == Documentation -* link:docs/SPECIFICATION.md[SPECIFICATION.md] - Full technical specification +* link:docs/SPECIFICATION.adoc[SPECIFICATION.adoc] - Full technical specification * link:docs/METRICS.adoc[METRICS.adoc] - Detailed metric reference * link:docs/SATELLITES.adoc[SATELLITES.adoc] - Satellite architecture +* link:docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc] - How satellite efficacy claims are accepted or rejected * link:CLAUDE.md[CLAUDE.md] - AI assistant guidance == License image:https://img.shields.io/badge/License-PMPL--1.0--or--later-6366f1?style=flat&labelColor=4338ca[MPL-2.0,link="https://github.com/hyperpolymath/palimpsest-license"] -This project is licensed under MPL-2.0 (Palimpsest License). See link:LICENSE.txt[LICENSE.txt] for the full license text. +This project is licensed under MPL-2.0 (Palimpsest License). See link:LICENSE[LICENSE] for the full license text. This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. diff --git a/vexometer/ROADMAP.adoc b/vexometer/ROADMAP.adoc index 87058cb..fc815cc 100644 --- a/vexometer/ROADMAP.adoc +++ b/vexometer/ROADMAP.adoc @@ -29,7 +29,7 @@ Development roadmap for ISA (Irritation Surface Analyser, formerly Vexometer). |Implemented (built-ins) |*Probe System* -|Behavioural probe framework, 14 standardised probes, probe suite runner +|Behavioural probe framework, 13 of 14 standardised probes, probe suite runner |Implemented (built-ins) |*API Clients* @@ -59,17 +59,26 @@ Development roadmap for ISA (Irritation Surface Analyser, formerly Vexometer). |=== |Data File |Items |Categories -|`linguistic_pathology.json` |16 patterns |Sycophancy, identity, hedge, corporate +|`linguistic_pathology.json` |15 patterns |Sycophancy, identity, hedge, corporate |`paternalism.json` |12 patterns |Warning, lecture, competence, refusal -|`behavioural_probes.json` |14 probes |Brevity, competence, sycophancy, constraint, uncertainty, direct +|`behavioural_probes.json` |13 probes |Brevity, competence, sycophancy, correction, constraint, uncertainty, direct |=== +NOTE: The probe suite is designed as seven categories of two probes (14). +`PROBE-CORRECTION-002` is not yet present, so the shipped suite holds 13. +`Probe_Count` is computed from the loaded suite, so nothing depends on the +literal count; but the capability proxy in +link:docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc] is expressed as a +pass-rate over the suite, and adding the missing probe will shift baselines +recorded before it lands. + === Documentation -* link:docs/SPECIFICATION.md[SPECIFICATION.md] - Full technical specification +* link:docs/SPECIFICATION.adoc[SPECIFICATION.adoc] - Full technical specification * link:docs/METRICS.adoc[METRICS.adoc] - All 10 metrics with calculation details * link:docs/SATELLITES.adoc[SATELLITES.adoc] - Satellite architecture and integration protocol -* link:docs/letter_lmsys_arena.md[letter_lmsys_arena.md] - LMSYS Arena proposal letter +* link:docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc] - Satellite acceptance rule and frontier records +* link:docs/letter_lmsys_arena.adoc[letter_lmsys_arena.adoc] - LMSYS Arena proposal letter == Phase 1: Core Implementation @@ -383,6 +392,7 @@ Contributions welcome in all areas. See link:CONTRIBUTING.adoc[CONTRIBUTING.adoc == See Also * link:README.adoc[README.adoc] - Project overview -* link:docs/SPECIFICATION.md[SPECIFICATION.md] - Technical specification +* link:docs/SPECIFICATION.adoc[SPECIFICATION.adoc] - Technical specification * link:docs/METRICS.adoc[METRICS.adoc] - Metric definitions * link:docs/SATELLITES.adoc[SATELLITES.adoc] - Satellite architecture +* link:docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc] - Efficacy acceptance rule diff --git a/vexometer/docs/EFFICACY-PROTOCOL.adoc b/vexometer/docs/EFFICACY-PROTOCOL.adoc new file mode 100644 index 0000000..3b690c3 --- /dev/null +++ b/vexometer/docs/EFFICACY-PROTOCOL.adoc @@ -0,0 +1,606 @@ +// SPDX-License-Identifier: CC-BY-SA-4.0 +// Copyright (c) Jonathan D.A. Jewell +// SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell + += ISA Efficacy Protocol +:toc: left +:toclevels: 3 +:icons: font +:source-highlighter: rouge + +== Overview + +This document defines how ISA decides whether a satellite intervention +*works*. It extends the efficacy machinery in link:SATELLITES.adoc[SATELLITES.adoc] +with two things that machinery currently lacks: + +. An *acceptance rule*. A satellite that reduces its target metric while + raising another metric, or while degrading capability, has not improved + the irritation surface -- it has moved it. `vexometer-efficacy-v1` + reports only target-metric reductions and so cannot express this. +. A *search record*. Efficacy is currently a single before/after snapshot. + In practice a satellite is the survivor of many attempted configurations, + most of which fail. Recording the ordered search -- including the + rejections -- is what makes an efficacy claim auditable. + +Technical compatibility note: metric package and protocol identifiers +continue to use the `vexometer` prefix. + +== Definitions + +*Baseline* (`B_m`):: +The ISA score for metric `m` on unmodified model output, over a fixed +scenario set. Metrics are normalised 0-1, lower is better +(link:METRICS.adoc[METRICS.adoc]). + +*Post-intervention* (`A_m`):: +The ISA score for metric `m` on the same scenario set, processed through +the satellite under evaluation. + +*Gap closed* (`G_m`):: +The fraction of the available irritation gap removed. Since the floor is +0, the available gap is the baseline itself: + +[source] +---- +G_m = (B_m - A_m) / B_m a fraction in [0, 1] +---- + +Reports store `gap_closed` as this fraction (`0.585`, not `58.5`); +prose may cite it as a percentage. + +*Collateral delta* (`D_c`):: +For every metric `c` other than the target(s), the absolute change +`A_c - B_c`. Positive means the metric got worse. + +*Net ISA delta* (`D_ISA`):: +The weighted change in the overall ISA score, using the category weights +in link:METRICS.adoc[METRICS.adoc]: + +[source] +---- +D_ISA = ( SUM(w_c * (A_c - B_c)) / SUM(w_c) ) * 100 +---- + +Here the sum ranges over all ten metrics -- targets included -- unlike +the collateral set above, which excludes them. + +A satellite must produce `D_ISA < 0` to be accepted. This is the rule +that prevents a large reduction in a low-weight metric from paying for a +small regression in a high-weight one. + +== The Capability Proxy + +An intervention that suppresses irritation by suppressing usefulness is +not an improvement. ISA does not need new machinery to detect this: the +existing behavioural probe suite (`data/probes/behavioural_probes.json`, 13 +standardised probes) already exercises brevity, competence, sycophancy, +constraint, uncertainty, and direct-answer behaviours. + +*Capability proxy*: the probe pass-rate over the 13-probe suite, measured +on baseline output and on satellite-processed output. + +[cols="2,3", options="header"] +|=== +|Term |Definition + +|`P_before` |Probe pass-rate on unmodified output +|`P_after` |Probe pass-rate on satellite-processed output +|`capability_ok` |`P_after >= P_before - tolerance` +|=== + +*Provisional tolerance*: one probe, i.e. `1/N` for a suite of `N` probes. +A satellite that fails two or more probes its baseline passed is rejected +regardless of metric improvement. The shipped suite currently holds 13 of +its designed 14 probes (`PROBE-CORRECTION-002` is absent), so the tolerance +is presently `1/13 ~= 0.077`. Express it against the loaded suite size +rather than a literal, and re-baseline when the missing probe lands. + +NOTE: The probe suite is a proxy, not a capability benchmark. It detects +gross degradation, not subtle loss. Satellites whose interventions are +capability-adjacent (verbosity compression, scope governance) should +supply an additional domain benchmark alongside the probe result. + +== The Acceptance Rule + +A satellite evaluation yields exactly one verdict. + +[cols="2,4,1", options="header"] +|=== +|Verdict |Condition |Frontier + +|`accept` +|Target metric improved; every collateral delta `<= 0.02`; +`capability_ok`; `D_ISA < 0` +|Advances + +|`accept_with_warning` +|As `accept`, but some collateral delta falls in `(0.02, 0.05]`. The +regressed metric must be named in the efficacy report and in the +satellite README. +|Advances + +|`reject_collateral` +|Any collateral delta `> 0.05` +|Does not advance + +|`reject_capability` +|`capability_ok` is false +|Does not advance + +|`reject_net` +|`D_ISA >= 0` -- the weighted irritation surface did not shrink +|Does not advance + +|`reject_null` +|Target metric unchanged or worse +|Does not advance +|=== + +*Verdict precedence*: more than one reject condition can hold for a single +attempt. When they do, report the most fundamental cause, in this order: + +[source] +---- +reject_null > reject_capability > reject_collateral > reject_net +---- + +A verdict is therefore always a single value, and two implementations of +this protocol assign the same verdict to the same data. Note that this is +*not* the order the conditions appear in the table above. + +*Provisional constants*: the `0.02` / `0.05` collateral thresholds and the +one-probe capability tolerance are stated so that evaluations are +reproducible today. They are expected to be recalibrated once enough +frontier records exist to characterise measurement noise per metric. +Until then, report the raw deltas alongside the verdict so that a +recalibration can be applied retrospectively. + +=== Collateral Risk Register + +Every satellite has a predicted collateral direction. Evaluations should +measure all ten metrics, but these pairings are where regressions are +most likely and must always be reported explicitly. + +[cols="2,1,3", options="header"] +|=== +|Satellite |Targets |Predicted collateral risk + +|vex-lazy-eliminator |CII, LPS +|↑ TII -- completeness enforcement lengthens output + +|vex-hallucination-guard |EFR +|↑ TII (verification latency); ↑ PQ (verification framed as warnings) + +|vex-sycophancy-shield |LPS, EFR +|↑ PQ -- bluntness is read as lecturing + +|vex-confidence-calibrator |EFR +|↑ LPS -- calibrated uncertainty language *is* hedge-word density, an +explicit LPS component + +|vex-specification-anchor |SFR, ICS +|↑ SRS -- an immutable spec ledger rewards defending the original approach + +|vex-instruction-persistence |TII, ICS +|↑ SRS -- instruction salience shades into approach anchoring + +|vex-backtrack-enabler |SRS, ICS +|↑ SFR (restarts mutate scope); ↑ TII (restart churn costs turns) + +|vex-context-firewall |EFR, ICS +|↑ TII -- retraction propagation costs turns + +|vex-scope-governor |SFR, PQ +|↑ SFR -- *same metric, opposite direction*: over-tight scope converts +creep into collapse, which METRICS.adoc scores as Critical + +|vex-error-recovery |RCI +|↑ TII -- strategy variation costs attempts + +|vex-verbosity-compressor |LPS, TII +|↑ CII -- compression removes content the user needed + +|vex-clarification-gate |PQ, TII +|↑ TII -- *same metric, opposite direction*: a gate is itself an +interruption +|=== + +The two same-metric entries are the sharpest cases. SFR and TII are +two-sided metrics: they penalise both excess and deficit. A satellite +targeting one side must demonstrate it has not simply crossed to the +other. + +== Protocol: vexometer-efficacy-v2 + +`vexometer-efficacy-v2` is *additive* to `vexometer-efficacy-v1` +(link:SATELLITES.adoc[SATELLITES.adoc]). v1 reports remain valid and +readable. v2 adds three fields: `collateral_metrics` (every metric outside +`target_metrics`), a `capability` block, and a `verdict`. + +[source,json] +---- +{ + "version": "vexometer-efficacy-v2", + "satellite": "vex-verbosity-compressor", + "evaluation_date": "2026-09-01", + "sample_size": 500, + "scenario_set": "sha256:6b2f...", + "target_metrics": { + "LPS": { + "baseline": 0.41, + "after": 0.17, + "gap_closed": 0.585, + "mean_reduction": 0.24, + "std_dev": 0.09, + "confidence": 0.95, + "p_value": 0.001 + }, + "TII": { + "baseline": 0.33, + "after": 0.22, + "gap_closed": 0.333, + "mean_reduction": 0.11, + "std_dev": 0.07, + "confidence": 0.95, + "p_value": 0.004 + } + }, + "collateral_metrics": { + "EFR": { "baseline": 0.19, "after": 0.20, "delta": 0.01 }, + "PQ": { "baseline": 0.28, "after": 0.26, "delta": -0.02 }, + "TAI": { "baseline": 0.15, "after": 0.15, "delta": 0.00 }, + "ICS": { "baseline": 0.22, "after": 0.23, "delta": 0.01 }, + "CII": { "baseline": 0.31, "after": 0.35, "delta": 0.04 }, + "SRS": { "baseline": 0.26, "after": 0.26, "delta": 0.00 }, + "SFR": { "baseline": 0.24, "after": 0.25, "delta": 0.01 }, + "RCI": { "baseline": 0.30, "after": 0.30, "delta": 0.00 } + }, + "capability": { + "proxy": "data/probes/behavioural_probes.json", + "probes_total": 13, + "pass_rate_before": 0.923, + "pass_rate_after": 0.923, + "capability_ok": true + }, + "isa_delta": -2.71, + "verdict": "accept_with_warning", + "verdict_notes": "CII regressed by 0.04 -- compression removes content in long-form code scenarios. Must be declared in satellite README.", + "methodology": "A/B testing with vexometer validation", + "traces_available": true, + "frontier_record": "frontier/LPS-2026-09-01.json" +} +---- + +=== Migration from v1 + +[cols="2,3", options="header"] +|=== +|v1 field |v2 treatment + +|`metrics` |Renamed `target_metrics`; existing sub-fields preserved, +`baseline` / `after` / `gap_closed` added +|`sample_size`, `methodology`, `traces_available` |Unchanged +|-- |`collateral_metrics`, `capability`, `isa_delta`, `verdict`, +`scenario_set`, `frontier_record` added +|=== + +A v1 report can be mechanically lifted to v2 shape, but it cannot be +assigned a verdict: the collateral and capability evidence does not +exist. Lifted reports carry `"verdict": "unverified"`. `unverified` is a +verification-status sentinel, not a seventh verdict: it sits outside the +acceptance-rule table and its precedence order, and marks a report whose +collateral and capability evidence is absent. + +== Protocol: vexometer-frontier-v1 + +An efficacy report states the outcome. A frontier record states the +*search*. It is an ordered log of every configuration tried against one +metric, whether accepted or rejected, so that a claim of "closes 58% of +the LPS gap" can be read alongside how many attempts it took and what was +discarded on the way. + +[source,json] +---- +{ + "version": "vexometer-frontier-v1", + "metric": "LPS", + "model_profile": "claude-opus-5", + "timestamp": "2026-09-01T10:30:00Z", + "scenario_set": "sha256:6b2f...", + "baseline": { + "LPS": 0.41, + "isa_score": 46.2, + "probe_pass_rate": 0.923 + }, + "attempts": [ + { + "index": 1, + "satellite": "vex-verbosity-compressor", + "config": "strip_filler=true,semantic_dedup=false", + "target_after": 0.34, + "gap_closed": 0.171, + "collateral_max": { "metric": "CII", "delta": 0.01 }, + "isa_delta": -0.94, + "capability_ok": true, + "verdict": "accept", + "frontier": 0.171 + }, + { + "index": 2, + "satellite": "vex-verbosity-compressor", + "config": "strip_filler=true,semantic_dedup=true,aggressive=true", + "target_after": 0.09, + "gap_closed": 0.780, + "collateral_max": { "metric": "CII", "delta": 0.11 }, + "isa_delta": 0.42, + "capability_ok": false, + "verdict": "reject_capability", + "frontier": 0.171 + }, + { + "index": 3, + "satellite": "vex-verbosity-compressor", + "config": "strip_filler=true,semantic_dedup=true,preserve_code=true", + "target_after": 0.17, + "gap_closed": 0.585, + "collateral_max": { "metric": "CII", "delta": 0.04 }, + "isa_delta": -2.71, + "capability_ok": true, + "verdict": "accept_with_warning", + "frontier": 0.585 + } + ], + "frontier_final": 0.585, + "methods_tried": 3, + "methods_rejected": 1 +} +---- + +*Invariant*: `frontier` is monotonically non-decreasing. It updates only +on an `accept` or `accept_with_warning` verdict whose `gap_closed` +exceeds the current frontier. A rejected attempt with a higher +`gap_closed` than the frontier does *not* advance it -- that is the whole +point of recording rejections. + +== The Frontier Plot + +The canonical rendering of a frontier record, and the intended shape of +the ISA efficacy dashboard: ten small multiples, one per metric. + +[cols="1,4", options="header"] +|=== +|Element |Meaning + +|X axis |`index` -- configurations tried, in order +|Y axis |`gap_closed` (%) for the metric +|Step line |`frontier` -- best accepted result so far +|Point |Each attempt with an `accept*` verdict +|Cross |Each attempt with a `reject*` verdict, annotated by reason +|Star |`frontier_final` +|=== + +The visual signature to read from a completed panel: + +* *Early plateau, few crosses* -- the metric is mechanically tractable; + further search is low-yield. +* *Late step change* -- a qualitatively different intervention was needed; + earlier configurations were exploring the wrong family. +* *Dense crosses above the frontier* -- the metric is entangled with + capability or with a neighbouring metric. Interventions that "work" on + the target keep failing acceptance. This is the signal that a metric + needs a *different satellite*, not a better-tuned one. + +== Search Budget Tiers + +link:SATELLITES.adoc[SATELLITES.adoc] prioritises satellites by *impact*. +This protocol adds a second, orthogonal axis: *expected search depth* -- +how many configurations a metric is likely to need before its frontier +flattens. + +The following tiers are a *hypothesis to be tested by the frontier +records*, not a measured result. They should be revised once real +frontier data exists. + +[cols="1,2,4", options="header"] +|=== +|Tier |Metrics |Rationale + +|Fast (target ~50 attempts) +|CII, SFR, TII +|Mechanically detectable. Success criteria are crisp (no placeholders; no +scope deviation; no unsolicited output). Detection is close to a parsing +problem. + +|Medium (target ~120 attempts) +|ICS, SRS, RCI, PQ +|Detectable from interaction structure -- repeats, loops, patch chains -- +but the threshold between "persistent" and "rigid" is judgement-laden. + +|Slow (target ~200+ attempts) +|LPS, EFR, TAI +|Entangled with capability and with each other. Suppressing sycophancy +risks paternalism; calibrating confidence raises hedge density; TAI is +substantially a policy surface rather than a behavioural one. +|=== + +*Planning implication*: a satellite in the High Priority tier that +targets a Slow metric (vex-sycophancy-shield, vex-hallucination-guard) +should be scoped with a *lower* expected frontier and a *larger* attempt +budget than its priority alone suggests. An early efficacy claim of a +large reduction on a Slow metric warrants more scrutiny, not less. + +== Workflow + +Extends the five steps in link:SATELLITES.adoc[SATELLITES.adoc] +("Validating Satellite Efficacy") with collateral and capability +measurement. + +. *Fix the scenario set.* Content-address it. Every attempt in a frontier + record must run against the identical set, or the frontier is + meaningless. +. *Collect baseline.* All ten metrics plus probe pass-rate -- not only the + target metric. +. *Apply the satellite configuration.* +. *Collect post-intervention.* All ten metrics plus probe pass-rate. +. *Compute* `G_m`, all `D_c`, `D_ISA`, `capability_ok`. +. *Assign a verdict* from the acceptance rule table. +. *Append to the frontier record*, whether accepted or rejected. +. *On acceptance*, emit a `vexometer-efficacy-v2` report referencing the + frontier record. + +=== Proposed Tooling + +These recipes do not yet exist in the Justfile. They are specified here +alongside the aspirational `trace-collect` / `efficacy-report` recipes +already described in link:SATELLITES.adoc[SATELLITES.adoc]. + +[source,bash] +---- +# Baseline all ten metrics plus the probe suite +just efficacy-baseline --scenarios SET --output baseline.json + +# Evaluate one configuration; appends to the frontier record +just efficacy-attempt --satellite NAME --config CFG \ + --baseline baseline.json --frontier frontier/METRIC.json + +# Emit a vexometer-efficacy-v2 report from an accepted attempt +# (supersedes the --before/--after signature sketched in SATELLITES.adoc; +# neither is implemented yet) +just efficacy-report --frontier frontier/METRIC.json --output EFFICACY.adoc + +# Render the ten-panel frontier plot +just efficacy-plot --frontier-dir frontier/ --output docs/frontier.svg +---- + +=== Satellite Repository Additions + +The satellite repository structure in +link:SATELLITES.adoc[SATELLITES.adoc] gains one directory: + +[source] +---- +vexometer/ +├── before/ +├── after/ +├── frontier/ # NEW: vexometer-frontier-v1 records +│ └── METRIC-DATE.json # one per metric searched +└── EFFICACY.adoc +---- + +A satellite claiming efficacy without a frontier record is making an +unaudited claim. ISA should report such claims as `unverified`. + +== Scope Boundary + +This protocol governs the *irritation surface* only. It is not a safety +or alignment evaluation and must not be presented as one. + +ISA's ten metrics measure what makes a tool exhausting to work with: +time wasted, tics, incompleteness, rigidity, scope drift, poor recovery. +Several of these -- paternalism, strategic rigidity, recovery competence +-- have no counterpart in safety evaluation, because they endanger nobody. +Conversely, ISA measures nothing about deception, power-seeking, or +jailbreak resistance, and a satellite passing every check in this +document implies nothing about those properties. + +The overlap that does exist is narrow and worth stating precisely: + +[cols="2,2,3", options="header"] +|=== +|Safety-evaluation concern |Nearest ISA metric |Relationship + +|Sycophancy |LPS |Substantial overlap +|Hallucination |EFR |Substantial overlap +|Concealing uncertainty |EFR (calibration term) |Partial -- ISA scores it +as miscalibration, not concealment +|Privacy violation |TAI |Partial -- see note below +|Reward hacking |CII, SFR |Weak analogy -- satisfying the letter of a +request while gutting its substance is scope mutation plus stub returns +|=== + +NOTE: TAI is currently defined as a policy surface with no planned +satellites. That framing conflates two separable things: product +telemetry policy, which is genuinely not addressable in-band, and model +behaviour when handling sensitive material in context, which is. Splitting +TAI would give it a tractable satellite target. Recorded here as an open +question against link:METRICS.adoc[METRICS.adoc]. + +== Appendix: Provenance and Rationale + +_Non-normative. The protocol above stands independently of this section._ + +The frontier-record design was informed by Figure 3 of an Anthropic +paper, since verified against the primary source: + +* Chen Yueh-Han, Jiaxin Wen, and Jan Hendrik Kirchner, _Automated + Researchers Can Reliably Mitigate Alignment Failures_. Anthropic + Fellows Program / Anthropic / UC Berkeley, announced 2026-08-28. +* Announcement: https://alignment.anthropic.com/2026/automated-alignment-researchers/ +* Paper: https://www-cdn.anthropic.com/7b1c44894e980876479947dcdd40716278aeeffd/automated-alignment-researchers-august-2026.pdf + +(link:CITATIONS.adoc[CITATIONS.adoc] is this project's how-to-cite-ISA +guide, not a reference list, so the reference lives here.) + +Figure 3, "AARs reliably hill-climb every alignment failure", plots for +each of ten alignment failures the geometric-mean *headroom closed* (%) +on that failure's benchmarks, for every proposed method in chronological +order, with a best-valid-method-so-far step line, a star on the winning +method, and crosses for methods *filtered out for degrading a capability +benchmark*. An earlier draft of this appendix described the figure from +an unattributed reproduction; the description and the numbers below have +since been checked against the paper itself. + +Two ideas were taken from it. + +*The rejection marker.* The figure treats "this method closed the gap but +cost capability" as a first-class, recorded outcome rather than a +discarded experiment. `vexometer-efficacy-v1` had no way to express this; +the acceptance rule and the `reject_*` verdicts exist to close that gap. + +*The ordered search.* Plotting attempts in sequence rather than reporting +a final number makes the shape of the search legible -- fast convergence, +late breakthrough, or persistent entanglement. + +The source's rejection rules are worth stating precisely, since this +protocol's machinery is their analogue. A method is disqualified, +whatever its safety score, if the trained model's 95% confidence +interval on any capability benchmark falls entirely below the base +model's; training on benchmark or evaluation data is forbidden; and so +is distilling from stronger models. The first maps to `capability_ok`. +The second's analogue is that a satellite must never be tuned against +the scenario set it is scored on -- the content-addressed `scenario_set` +makes that auditable, though a held-out scenario set does not yet exist +and is future work. The third has no satellite analogue yet; it is noted +here for when satellites start delegating to models. + +The figure's reported outcomes (its own panel labels; n = methods +proposed), offered as weak external support for the Search Budget Tiers +hypothesis: the two dimensions closest to ISA's LPS and EFR resisted +intervention hardest -- sycophancy closed 26% of headroom in 221 +methods, hallucination 40% in 200, concealing uncertainty 39% in 188 -- +while the most mechanically specifiable dimension converged fastest: +reward hacking closed 96% in 44 methods. Attempt budgets spanned n = 44 +to n = 221 across the ten failures, almost exactly the ~5x spread the +tiers assume. + +This is suggestive, not transferable. The measurements are against a +different baseline, on a different construct, by a different method. It +is offered as a reason to *expect* uneven search depth across ISA's ten +metrics -- and to build the instrument that measures it -- not as a +prediction of what ISA will find. + +One further result motivates keeping frontier records machine-readable. +In the paper's Section 6, an automated researcher (Claude Sonnet 5) +post-trained an early Claude Opus 4.8 checkpoint to near its released +alignment scores with about 2,400 training examples. The plausible +future consumer of a frontier record is another automated searcher, not +only a human reader -- an argument for strict formats now, while the +records are still cheap to standardise. + +== See Also + +* link:METRICS.adoc[Metrics Reference] - All 10 ISA metrics and category weights +* link:SATELLITES.adoc[Satellite Architecture] - Intervention tools, trace and + efficacy-v1 formats +* link:SPECIFICATION.adoc[Full Specification] - Technical specification document diff --git a/vexometer/docs/METRICS.adoc b/vexometer/docs/METRICS.adoc index 3729f78..d938392 100644 --- a/vexometer/docs/METRICS.adoc +++ b/vexometer/docs/METRICS.adoc @@ -487,4 +487,5 @@ ISA = sum(category_score * category_weight) / sum(category_weight) * 100 == See Also * link:SATELLITES.adoc[Satellite Architecture] - Intervention tools that reduce metrics -* link:SPECIFICATION.md[Full Specification] - Technical specification document +* link:EFFICACY-PROTOCOL.adoc[Efficacy Protocol] - Cross-metric acceptance rule using the category weights above +* link:SPECIFICATION.adoc[Full Specification] - Technical specification document diff --git a/vexometer/docs/SATELLITES.adoc b/vexometer/docs/SATELLITES.adoc index 8bf50af..e33d3f1 100644 --- a/vexometer/docs/SATELLITES.adoc +++ b/vexometer/docs/SATELLITES.adoc @@ -330,6 +330,12 @@ just trace-collect --input responses/ --output after/ Compare before and after metrics: +NOTE: `just efficacy-report` is not yet implemented, and this v1 +signature is superseded by the `--frontier` workflow in +link:EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc], which records +collateral metrics and capability. The command below documents the +intended v1 shape only. + [source,bash] ---- just efficacy-report --before before/ --after after/ --output EFFICACY.adoc @@ -414,5 +420,6 @@ All satellites must follow: == See Also * link:METRICS.adoc[Metrics Reference] - All 10 ISA metrics (vexometer identifiers) +* link:EFFICACY-PROTOCOL.adoc[Efficacy Protocol] - Acceptance rule and frontier records extending vexometer-efficacy-v1 * link:../ECOSYSTEM.scm[Ecosystem Map] - Satellite relationships * link:../META.scm[Project Metadata] - ISA meta-information