feat(memory): measure continual learning against stateless runs

[drewstone]

What changed

• compare matched stateful and stateless memory runs across ordered tasks
• report post-first-step gain, explicitly labeled transfer, and repeated-probe forgetting
• average repetitions and candidates within independent sequences before paired statistics
• bind arms to immutable candidate, executor, task, artifact, and scoring-evidence identities
• share one cost limit and preserve safe cancellation, cleanup, recovery, and resume state

Proof

• pnpm test: 545 passed, 12 skipped
• pnpm typecheck
• pnpm lint: 202 files
• pnpm build
• pnpm verify:package: clean packed install and imports
• merged and checked against current main

Limit

No paid Mem0, Graphiti, or Neo4j run is claimed here. A real learning result requires independent runs with both arm orders.

[tangletools]

✅ Auto-approved drewstone PR — 5b64e68b

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-07-30T00:27:58Z}

[tangletools]

🟡 Value Audit — sound-with-nits

Verdict	sound-with-nits
Concerns	1 (1 weak-concern)
Heuristic	0.0s
Duplication	0.0s
Interrogation	264.9s (2 bridge agents)
Total	264.9s

💰 Value — sound-with-nits

Adds a rigorously-evidenced stateful-vs-stateless continual-learning comparison instrument built on existing campaign/lifecycle primitives; sound architecture, one optional reuse of agent-eval's comparePairedArms for the cell-level gain.

• What it does: Adds runAgentMemoryLearningExperiment, which runs two matched 'arms' of the existing runAgentMemoryExperiment — one stateful (retains declared scopes across ordered steps) and one stateless (clears every declared scope before each step after the first) — under one shared cost ledger/ceiling, then pairs them cell-by-cell and probe-by-probe. It reports (a) post-first-step 'gain' = stateful minus sta
• Goals it achieves: Establish whether retained memory actually helps across ordered tasks, with statistical rigor (paired bootstrap), auditable evidence (every probe score reproducible from a saved content-addressed input), and provably-comparable arms (non-equivalent arms fail comparison — same splitDigest, candidateRefs, executionRef, comparisonRef). Secondary goals achieved incidentally and correctly: safe cancell
• Assessment: A coherent, deep extension that stays in the codebase's grain. It reuses runCampaign, pairedBootstrap, runBoundedMemoryLifecycle, createMemoryExecutionPool, the cost ledger, run-control leases, and the recovery/attempt-log machinery rather than reinventing them (learning.ts is a 111-line orchestrator calling runAgentMemoryExperiment twice + compareAgentMemoryLearning once). The comparison-ref / ev
• Better / existing approach: Searched src/optimization, src/memory, src/release, and @tangle-network/agent-eval for existing two-run comparison. agent-eval DOES export a generic comparePairedArms/pairArms (projects records to PairedArmRow, does identity matching + McNemar + pairedBootstrap), and pairRunRecords (RunRecord-specific). pairRunRecords is genuinely inapplicable (memory experiments emit CampaignResult<AgentMemorySeq
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound

A rigorous stateful-vs-stateless continual-learning measurement layer that composes the existing memory experiment primitive in the codebase's established grain, reachable from the public package surface and documented, ahead of its first in-package caller but answering a distinct question from the

• Integration: Both new exports (runAgentMemoryLearningExperiment, compareAgentMemoryLearning) are wired into the public package surface: src/memory/experiment.ts:6-9 re-exports them, reachable via @tangle-network/agent-knowledge/memory (package.json:32-34). They compose the existing runAgentMemoryExperiment primitive (learning.ts:68 calls it twice with different memoryMode). The standalone `compar
• Fit with existing patterns: Fits the established composition pattern precisely. The improvement loop (improvement/run.ts:131-184) also composes runAgentMemoryExperiment underneath via evaluateMemoryCandidate; the learning experiment is the same pattern at a different axis (stateful vs stateless instead of config vs config). The memoryMode field is optional, defaults to 'stateful' (comparison-ref.ts:25-30), so exi
• Real-world viability: Holds up well. The pairing logic (learning-pairs.ts:59-141) is exhaustive: it validates that both arms share identical split identity, candidate refs, execution refs, seeds, reps, and comparisonRef, and that manifests differ (proving memoryMode is actually distinguished), throwing on every mismatch rather than silently degrading. Stateless mode correctly requires adapter.clear (`cell.ts:170-17
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

💰 Value Audit

🟡 Cell-level gain matching reimplements agent-eval's comparePairedArms [duplication] ``

learning-pairs.ts:35-57 identity-keys cells on [candidateId, sequenceId, rep, seed] and fail-loud on unmatched/duplicate, and learning-metrics.ts:48-52,260-276 run the headline pairedBootstrap on cell rewards. agent-eval exports comparePairedArms/pairArms which does exactly this (identity matching + per-metric pairedBootstrap). The headline cell-gain could delegate to comparePairedArms by projecting each cell to a PairedArmRow{pairKey=candidateId:sequenceId, repKey=rep:seed, arm=memoryMode, metr

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260730T003258Z}

[tangletools]

✅ Auto-approved drewstone PR — 077017c2

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-07-30T00:47:57Z}

[tangletools]

🟢 Value Audit — sound

Verdict	sound
Concerns	0 (none)
Heuristic	0.0s
Duplication	0.0s
Interrogation	116.6s (2 bridge agents)
Total	116.6s

💰 Value — sound

Adds a rigorous stateful-vs-stateless paired learning experiment that composes existing run/paired-comparison primitives to measure whether memory persistence confers a real advantage; ships as-is.

• What it does: Adds a memoryMode ('stateful'|'stateless') to the existing runAgentMemoryExperiment — stateless clears all declared sequence scopes before each step after the first. On top of that, runAgentMemoryLearningExperiment runs both arms of the same candidate under one shared cost ledger (counterbalanced order), and compareAgentMemoryLearning pairs them cell-by-cell and probe-by-probe to report: p
• Goals it achieves: Answer the scientific question 'does persistent memory actually help?' with valid paired statistics and tamper-evident provenance. The gain metric isolates the persistence effect by differencing matched stateful/stateless reward on post-first-step probes; pre-treatment balance catches arm drift; transfer and forgetting are measured only on explicitly-labeled probes (no inference from prompt text).
• Assessment: This is a high-quality, well-architected change in the grain of the codebase. It composes existing primitives rather than reinventing: the learning experiment calls the existing runAgentMemoryExperiment twice (no fork), the paired comparison reuses comparePairedArms/pairArms from agent-eval, and the latest branch commit (077017c) actively consolidated four hand-rolled pairedBootstrap sites
• Better / existing approach: none — this is the right approach. Searched for any existing stateful/stateless, learning-gain, forgetting, or transfer capability via grep across src/ and found none outside the new files. The existing improvement-loop comparison (src/memory/improvement/evaluation.ts loadFinalEvaluation) answers a different question — ranking different candidate configurations on stored artifacts — and lacks
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound

A coherent stateful-vs-stateless memory measurement layer built directly on the existing experiment pipeline, reusing agent-eval's paired-bootstrap and enforcing strict arm-matching; it answers a distinct question no existing capability covers.

• Integration: Fully reachable. runAgentMemoryLearningExperiment and compareAgentMemoryLearning are exported from the public package surface (src/memory/experiment.ts:7-9), documented in README.md:242-271, and tested (tests/memory/experiment-learning.test.ts, 506 lines). No internal caller invokes them yet — they are a public measurement API for external consumers, the normal 'capability ahead of first calle
• Fit with existing patterns: Fits the codebase grain precisely. The stateful/stateless distinction is a memoryMode parameter threaded through the existing cell/run/scenario pipeline (src/memory/experiment/run.ts:55,170), not a parallel system. The stateless arm reuses the existing clearSequenceScopes + adapter.clear machinery already present for cleanup (src/memory/experiment/cell.ts:189-194). It reuses `comparePairedAr
• Real-world viability: Robust beyond the happy path. One shared cost ledger across both arms with enforced ceiling equality (src/memory/experiment/learning.ts:49-53). Resume support (rerun same options/dir). Signal cancellation threaded through every step with bounded provider operations via runBoundedMemoryLifecycle. Arm matching is cryptographically strict: comparison refs must be identical, split identities must matc
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

No concerns — sound change, no better or existing approach found. ✅

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260730T005019Z}