audit: finish Eval isolation, statistics, exports, docs, and release proof

[drewstone]

Goal

Finish the five remaining parts of the agent-eval audit. Prior findings and receipts remain in the issue comments.

Already Complete

• Final-case isolation in campaign comparison and official GEPA paths
• Paired bootstrap intervals, exact sign tests, multiplicity handling, ties, and the mean regression
• Official GEPA 0.1.4 integration, source pin, failure handling, and final recomputation
• Removal of old lineage, hand-written optimizer families, shadow-probe, and belief-state
• Current npm release archive checks and the packed README quickstart

Completion Criteria

1. Protect all proposal inputs. Apply the judge and final-case data check inside runOptimization. Add integration tests for findings, reports, datasets, trace text, history metadata, and error text.
2. Finish statistics review. Audit all 21 Math.random references across 16 source files, cite each formula and consumer, add missing hand fixtures, and prove sequential state reconstruction after process restart.
3. Classify the public API. Review all 25 package subpaths and 278 root exports. For each non-type export, name a production consumer, a tracked planned consumer, or delete it.
4. Classify tests and docs. Resolve 41 no-throw assertions, 5 module mocks, and 4 conditional skips. Verify or explicitly mark the 103 runnable-looking code fences across 43 Markdown files.
5. Prove release history. Correct the missing 87c02e6 entry for 0.130.1, compare the last five PyPI archives with their source tags, and post the final five highest risks with receipts.

Separate Work

• Exact packed testing across Eval, Knowledge, and Runtime is tracked in agent-runtime #628.
• TypeScript 7 documentation support is tracked in agent-runtime #625.
• Official GEPA integration is complete and is not part of the remaining work.

[drewstone]

Audit disposition against main@9332ebc7, with PR #446 reviewed separately: keep this issue open.

Proven complete

• final-case isolation in the campaign loop, comparison preset, and official GEPA path
• paired bootstrap intervals, exact sign tests, multiplicity handling, and the mean-versus-median tie regression
• sequential stopping behavior within one process
• official GEPA payload isolation, source pinning, failure handling, and final recomputation
• current npm release artifacts match their last five tags
• README quickstart executes from the packed package

Obsolete after current cleanup

• the historical rising-bar path
• the hand-written GEPA, ACE, HALO, and FAPO families
• runLineageLoop and its old lineage implementation
• the deleted Runtime self-improving-supervisor failure

Confirmed live defects

1. High: assertNoJudgeVerdict is exported and documented but has zero production call sites. runOptimization can pass opaque analyst findings directly to a proposer without applying the steering check.
2. High: Eval exposes only CampaignCostMeter.runPaidCall, while Runtime reportLoopUsage requires observe and observeTokens. The public example therefore describes an object that current Eval does not provide, and no cross-package compile test catches it.
3. Medium: two steering optimizer classes remain public with no production caller across Eval, Runtime, Knowledge, or Loops.
4. Medium: changelog entries for 0.130.1 and 0.130.0 omit shipped commits 87c02e6 and 25ce61b.

Remaining completion work

• add adversarial steering tests for findings, reports, datasets, trace text, prior-candidate metadata, and error text
• prove sequential reconstruction after process restart
• audit every random source and statistical consumer, including formula references and hand fixtures
• compile current Eval against current Runtime in CI
• classify the 17 no-throw tests, 5 module mocks, and 9 skip sites
• complete the exported-API and documentation claim census
• compare the last five PyPI archives with their source tags

PR #446 adds the profile measurement path and fixes its execution safety, concurrency accounting, and paired-precision decision. It does not close the items above.

[drewstone]

Disposition on main@b75838a and npm 0.134.0: keep open.

Completion criterion 1 is now done by #467. runOptimization accepts only ProposalFinding[], validates initial, baseline-analysis, generation-analysis, and proposal-time findings, freezes the proposal context, and has integration coverage for rejected unclassified inputs plus accepted search/production origins. Runtime #643 adopted the same contract.

The remaining work is criteria 2 through 5: random-source and statistical-consumer review including restart reconstruction; public export classification; no-throw/mock/skip and runnable-doc classification; and release-history/PyPI proof. Do not close this issue on the proposal isolation release alone.

[drewstone]

Disposition against origin/main@ed3487799fab93184617339b2e7f65c1e0d315b2, npm @tangle-network/agent-eval@0.135.1, and PyPI agent-eval-rpc==0.135.1: keep open.

Done: criterion 1 shipped in #467 (a75dbabc222eec751debc5c56f13d0e23a9f9e71). runOptimization calls assertProposalFindings for initial (run-optimization.ts:181), baseline-analysis (:284), proposal-time (:307), and generation-analysis (:478) inputs; adversarial coverage is in run-optimization-proposal-findings.test.ts. PR proof was 4,106 passed, 3 skipped across 309 files, plus successful typecheck, build, and packed-package checks.

Remaining: criteria 2 through 5. Current main has 20 Math.random references across 16 source files; sequential state at src/campaign/gates/sequential.ts:197-263 can be read but not restored and has no restart test. The package still publishes 25 subpaths without the required consumer/planned-consumer/delete classification. The requested no-throw, module-mock, conditional-skip, and 103-fence classification is not present. CHANGELOG.md still omits 87c02e6, and neither the five-PyPI-archive comparison nor the final five risks has been posted.

[drewstone]

Disposition after npm and PyPI 0.135.2 at 1243ca2: keep open.

Release 0.135.2 fixed paired promotion interval semantics in #477 through #480. It did not complete criteria 2 through 5. Fresh source checks still find 20 Math.random references across 16 files; makeStream exposes state() but has no restart/restore input; package.json still publishes 25 export keys; and CHANGELOG.md still has the 0.130.1 and 0.130.0 headings without the missing commit receipts named in this issue. No completed export-consumer classification, test/doc classification, five-PyPI-archive comparison, or final five-risk report is present.

Criterion 1 remains complete. Criteria 2 through 5 remain the exact closure work.