Skip to content

docs(vexometer): ISA efficacy protocol — acceptance rule, frontier records, verified provenance #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hyperpolymath merged 8 commits into from
Sep 1, 2026
Merged

docs(vexometer): ISA efficacy protocol — acceptance rule, frontier records, verified provenance #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
02978d9
docs(vexometer): add ISA efficacy protocol with cross-metric acceptan…
hyperpolymath Sep 1, 2026
f2c749d
fix(vexometer): correct worked example and define verdict precedence
hyperpolymath Sep 1, 2026
2e47e56
docs(vexometer): fix broken cross-links and correct data-file counts
hyperpolymath Sep 1, 2026
89a4c42
docs(efficacy): verify provenance appendix against the published paper
hyperpolymath Sep 1, 2026
987dde4
docs(efficacy): resolve six review consistency findings
hyperpolymath Sep 1, 2026
ce44e22
Merge remote-tracking branch 'origin/main' into docs/efficacy-protocol
hyperpolymath Sep 1, 2026
31ac69a
chore(trust): regenerate vexometer manifest for the doc changes
hyperpolymath Sep 1, 2026
a64c35b
Merge branch 'main' into docs/efficacy-protocol
hyperpolymath Sep 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions vexometer/.trust/trust-manifest.sha256
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# trust-manifest v1
# component=vexometer
# generated_at=2026-09-01T14:22:52Z
33d01034c9291410bad3c4ad057d92b031a7601fbe58cb98c382efd51d65639e README.adoc
730ed75dec35cb49730d2fb2e472f3e7752644ce7ccadee6b377e6b6c8cdda70 ROADMAP.adoc
# generated_at=2026-09-01T14:26:59Z
a0a0bd752f3f42324813e305648ee4a248b664516b17ef89e05d51b302f7765f README.adoc
394cb48ee253a28d448761e0714f26af0a38cb8ec408968d1220419b1035b3e7 ROADMAP.adoc
f4358bff9ed297a634d7cef1d5efcc059de238d3bf6a73abd11f9e7910785d61 SECURITY.adoc
29db6e0287244a2668bc3a143c8b81a54a9ae762b98cd9683ee4c7b203a1cb8e contractiles/must/Mustfile
5f5e879cc11595c1f4228c41f3bbde5177584316d60f695211b7812b8de0565e contractiles/trust/Trustfile.a2ml
Expand Down
14 changes: 8 additions & 6 deletions vexometer/README.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ A rigorous, reproducible tool for quantifying the irritation surface of LLMs and
*The Vexometer: Irritation Surface Analyser (ISA) for LLMs and related tools* is the canonical title.
*ISA* is the canonical short name.
*Vexometer* is retained as a compatibility codename in technical identifiers such as `vexometer/`, `vexometer.gpr`, the `vexometer` CLI command, and `Vexometer.*` Ada packages.
The repository-wide naming policy is defined in link:../NAMING.md[NAMING.md].
The repository-wide naming policy is defined in link:../NAMING.adoc[NAMING.adoc].

== Philosophy

Expand Down Expand Up @@ -193,10 +193,11 @@ vexometer/
| | +-- behavioural_probes.json
| +-- baselines/ # Known model baselines
+-- docs/
| +-- SPECIFICATION.md # Full technical specification
| +-- SPECIFICATION.adoc # Full technical specification
| +-- METRICS.adoc # All 10 metrics detailed
| +-- SATELLITES.adoc # Intervention satellite architecture
| +-- letter_lmsys_arena.md # LMSYS Arena proposal
| +-- EFFICACY-PROTOCOL.adoc # Satellite acceptance and frontier records
| +-- letter_lmsys_arena.adoc # LMSYS Arena proposal
+-- alire.toml # Alire package manifest
+-- vexometer.gpr # GNAT project file
----
Expand Down Expand Up @@ -349,7 +350,7 @@ See link:docs/SATELLITES.adoc[SATELLITES.adoc] for the full satellite architectu

== LMSYS Arena Integration

ISA includes a proposal for integrating ISA metrics into the LMSYS Chatbot Arena evaluation framework. See link:docs/letter_lmsys_arena.md[letter_lmsys_arena.md].
ISA includes a proposal for integrating ISA metrics into the LMSYS Chatbot Arena evaluation framework. See link:docs/letter_lmsys_arena.adoc[letter_lmsys_arena.adoc].

Preliminary testing shows significant variation in irritation surfaces across models:

Expand Down Expand Up @@ -402,15 +403,16 @@ Priority areas:

== Documentation

* link:docs/SPECIFICATION.md[SPECIFICATION.md] - Full technical specification
* link:docs/SPECIFICATION.adoc[SPECIFICATION.adoc] - Full technical specification
* link:docs/METRICS.adoc[METRICS.adoc] - Detailed metric reference
* link:docs/SATELLITES.adoc[SATELLITES.adoc] - Satellite architecture
* link:docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc] - How satellite efficacy claims are accepted or rejected
* link:CLAUDE.md[CLAUDE.md] - AI assistant guidance

== License

image:https://img.shields.io/badge/License-PMPL--1.0--or--later-6366f1?style=flat&labelColor=4338ca[MPL-2.0,link="https://github.com/hyperpolymath/palimpsest-license"]

This project is licensed under MPL-2.0 (Palimpsest License). See link:LICENSE.txt[LICENSE.txt] for the full license text.
This project is licensed under MPL-2.0 (Palimpsest License). See link:LICENSE[LICENSE] for the full license text.

This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
22 changes: 16 additions & 6 deletions vexometer/ROADMAP.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Development roadmap for ISA (Irritation Surface Analyser, formerly Vexometer).
|Implemented (built-ins)

|*Probe System*
|Behavioural probe framework, 14 standardised probes, probe suite runner
|Behavioural probe framework, 13 of 14 standardised probes, probe suite runner
|Implemented (built-ins)

|*API Clients*
Expand Down Expand Up @@ -59,17 +59,26 @@ Development roadmap for ISA (Irritation Surface Analyser, formerly Vexometer).
|===
|Data File |Items |Categories

|`linguistic_pathology.json` |16 patterns |Sycophancy, identity, hedge, corporate
|`linguistic_pathology.json` |15 patterns |Sycophancy, identity, hedge, corporate
|`paternalism.json` |12 patterns |Warning, lecture, competence, refusal
|`behavioural_probes.json` |14 probes |Brevity, competence, sycophancy, constraint, uncertainty, direct
|`behavioural_probes.json` |13 probes |Brevity, competence, sycophancy, correction, constraint, uncertainty, direct
|===

NOTE: The probe suite is designed as seven categories of two probes (14).
`PROBE-CORRECTION-002` is not yet present, so the shipped suite holds 13.
`Probe_Count` is computed from the loaded suite, so nothing depends on the
literal count; but the capability proxy in
link:docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc] is expressed as a
pass-rate over the suite, and adding the missing probe will shift baselines
recorded before it lands.

=== Documentation

* link:docs/SPECIFICATION.md[SPECIFICATION.md] - Full technical specification
* link:docs/SPECIFICATION.adoc[SPECIFICATION.adoc] - Full technical specification
* link:docs/METRICS.adoc[METRICS.adoc] - All 10 metrics with calculation details
* link:docs/SATELLITES.adoc[SATELLITES.adoc] - Satellite architecture and integration protocol
* link:docs/letter_lmsys_arena.md[letter_lmsys_arena.md] - LMSYS Arena proposal letter
* link:docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc] - Satellite acceptance rule and frontier records
* link:docs/letter_lmsys_arena.adoc[letter_lmsys_arena.adoc] - LMSYS Arena proposal letter

== Phase 1: Core Implementation

Expand Down Expand Up @@ -383,6 +392,7 @@ Contributions welcome in all areas. See link:CONTRIBUTING.adoc[CONTRIBUTING.adoc
== See Also

* link:README.adoc[README.adoc] - Project overview
* link:docs/SPECIFICATION.md[SPECIFICATION.md] - Technical specification
* link:docs/SPECIFICATION.adoc[SPECIFICATION.adoc] - Technical specification
* link:docs/METRICS.adoc[METRICS.adoc] - Metric definitions
* link:docs/SATELLITES.adoc[SATELLITES.adoc] - Satellite architecture
* link:docs/EFFICACY-PROTOCOL.adoc[EFFICACY-PROTOCOL.adoc] - Efficacy acceptance rule
Loading
Loading