feat(agent): A4 compaction phase 2 — compaction checkpoint carrier + prefer-checkpoint seed (plan #949) - #954
feat(agent): A4 compaction phase 2 — compaction checkpoint carrier + prefer-checkpoint seed (plan #949)#954btipling wants to merge 5 commits into
Conversation
…prefer-checkpoint seed (plan #949) - NEW cap COMPACTION_CHECKPOINT_MAX_BYTES (1 MiB, generous; no existing cap changed → no human gate) - sessionStore: reserved meta key compactionPointer (4th worker-authored sibling) + copyForwardCompactionPointer + validateMeta drop-to-unset arm; wired into both store upsertEnvelopes - workerMetaOverlay: compactionPointer in WORKER_META_KEYS + sanitize arm (distinct from the B6 checkpointPointer sibling) - persistStep fold carries compactionCheckpoint; PersistStepResult exposes compactionPointer - turnPersistSeam: checkpoint written as its OWN Blob object under COMPACTION_CHECKPOINT_MAX_BYTES via writeSegment; fail-closed compaction_write_failed; patch.compactionPointer; prior pointer survives via copy-forward (durable, no volatility) - /api/turns seed: prefer compactionPointer (re-validate shape + re-pair retainedTail + renderSummaryRow first), fall back to modelMessagesPointer, then legacy promptHistory sidecar (locked fallback chain); #937 fail-closed 503 now covers both bound pointers - cloudMetaFor/trimForCloudPut never emit compactionPointer (worker-authored; host omit copy-forwards) - tests: persist seam rows 1a–1c, sessionStore sanitize/copy-forward, workerMetaOverlay sanitize, route seed rows 2a–2h, sessionRepository host-omit; full suite 176 files / 3450 tests green Closes #949. Refs #947 (parent), #552 (source). Phase 1: #953.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
btipling
left a comment
There was a problem hiding this comment.
Adversarial review — PR #954
Verdict: PASS WITH NOTES
Repo: btipling/invincible
Scope: main ← plan/a4-compaction-p2 (5a3d5a3) · 13 files · A4 compaction phase 2 carrier + prefer-checkpoint seed
Lenses run: L1, L2, L5, L6, L8 (skip: L3 no DOM/Wasm/chat surface; L4 no CI/artifact/workflow change; L7 no host/config bind; L9 no UI)
AGENTS.md read: yes (also docs/feature-divide.md ownership of /api/turns; persist-step/API surface checked against SECURITY.md — no secret/runner reach)
Findings
| Sev | Lens | Finding | Break scenario | Refutation attempt | Confidence |
|---|---|---|---|---|---|
| Minor | L1 | POST /api/turns shape-checks the checkpoint then re-pairs only retainedTail via buildModelMessages. It does not re-run buildCheckpoint / boundSummary / boundFilesTouched on planted summary / filesTouched. Phase-1 caps COMPACTION_SUMMARY_MAX_CHARS (8 000 code points) and COMPACTION_FILES_TOUCHED_MAX (256, last-occurrence + omitted-count honesty) are write-time-only. renderSummaryRow sanitizes control-char paths but does not cap count or summary length. Path: app/api/turns/route.ts seed block; contrast lib/agent/compaction.ts buildCheckpoint JSDoc (“phase-2 read seam re-validates + re-pairs on read”) and COMPACTION_CHECKPOINT_MAX_BYTES comment (“the route re-validates + re-pairs on read”). |
1. Same-session bound Blob whose body is {summary: "🙂"×8001, filesTouched: 257 unique paths, retainedTail: [user row]} and meta.compactionPointer pointing at it (hostile client PUT of a valid opaque id, or a Phase-3 producer that skips buildCheckpoint; persist writes JSON.stringify(fold.compactionCheckpoint) with no builder). 2. Next POST /api/turns accepts the shape, renders the raw summary + all 257 paths into the honesty user row, then #944 token-trims. 3. Default/large window (CONTEXT_WINDOW_DEFAULT_TOKENS − reserve ≈ 16k tokens; 8k chars is ~2k tokens) keeps the over-cap row. The cut’s retained tail then yields token budget to a summary that Phase 1 forbade. Marker … (N earlier paths omitted) is missing. |
Defender: persist is a dumb carrier; Phase 3 will call buildCheckpoint; 1 MiB writeSegment fail-closes; #944 trim drops the summary first on an extreme miss; isObjectIdBoundTo blocks foreign blobs; shape check rejects non-checkpoints; renderSummaryRow already drops control-char paths. That fails: (a) the Phase-1/Phase-2 lock is read-side re-validation, matching the #937 planted-projection rebuild they already do for the tail; (b) persist does not call buildCheckpoint; (c) token trim is the wrong rail (window-derived, not 8 000 chars / 256 paths); (d) no test plants an over-cap summary/files list. |
high |
| Nit | L6 | lib/sessions/sessionStore.test.ts duplicates the #949 copyForwardCompactionPointer unit and the upsertEnvelope copy-forward integration (two near-identical it blocks each). |
A later edit updates one copy and leaves the other asserting the old contract → false green or a confusing fail. | Defender: extra coverage is harmless. True, but the second pair adds no assertion the first lacks (the first upsert case already locks “durable, no volatility”). | high |
Residual risk
Phase 3 (#950) is still the producer — this PR will not compact a live session. A bound pointer whose Blob is missing still 503s only when the host also omitted promptHistory (locked #937 fail-closed, shared across both pointers). GET overlay / host sidecar-stop still keys solely on modelMessagesPointer; compaction is model-facing seed, not a paint/sidecar signal — correct for this phase, but a compaction-only fold with no modelMessagesPointer keeps the host sending the legacy sidecar until Phase 3 also writes the sibling. B7/B8 partial-commit (compaction Blob written, overlay lww_conflict) can orphan a checkpoint object the same way #936 / B6 already can; documented, not new. 503 copy still says “model-messages seed” when the miss was compactionPointer.
Merge guidance
PASS WITH NOTES: safe to merge from this attack once the Minor is taken (or explicitly deferred to Phase 3 with a test that the read path currently does not bound). Nit is optional. Do not treat “Phase 3 will call buildCheckpoint” as the read-side lock — that is the write producer; the planted-blob case is this PR’s seed.
What was not attacked
Live Workflows/'use step' serialization of a 1 MiB fold; prod Redis/Blob; DO runner; Gateway; Wasm paint; Phase 3 trigger/summarizer (#950); parent #947 close-out.
#954) Planted summary/filesTouched bypassed COMPACTION_SUMMARY_MAX_CHARS and COMPACTION_FILES_TOUCHED_MAX because the route only re-paired the tail. Drop the duplicate sessionStore copy-forward tests.
btipling
left a comment
There was a problem hiding this comment.
Adversarial review — PR #954
Verdict: CONCERNS
Repo: btipling/invincible
Scope: main ← plan/a4-compaction-p2 (46a8d48) · 13 files · A4 compaction phase 2 carrier + prefer-checkpoint seed (includes follow-up fix(agent): re-run buildCheckpoint on compaction seed read)
Lenses run: L1, L2, L5, L6, L8 (skip: L3 no DOM/Wasm/chat surface; L4 no CI/artifact/workflow change; L7 no host/config bind; L9 no UI)
AGENTS.md read: yes (docs/feature-divide.md ownership of /api/turns; persist-step/API surface checked against SECURITY.md — no secret/runner reach)
Findings
| Sev | Lens | Finding | Break scenario | Refutation attempt | Confidence |
|---|---|---|---|---|---|
| Major | L1 | POST /api/turns re-runs buildCheckpoint on every well-shaped checkpoint (the #954 follow-up). buildCheckpoint is not idempotent once it has baked the files-omitted honesty line into summary. boundSummary then treats that suffix as overflow, drops Files read/modified: … (N earlier paths omitted), and can stamp a lying … [summary truncated] marker on a summary that was already at-cap. Route comment claims the bake is idempotent after buildModelMessages. Path: lib/agent/compaction.ts buildCheckpoint / boundSummary; app/api/turns/route.ts seed block. |
1. Phase 3 (or any producer) calls buildCheckpoint({summary: "x"×8000, filesTouched: 300 unique paths}, tail) and persist writes that Blob. Stored summary is x×8000 + omitted line (~8050 UTF-16 units), filesTouched is the newest 256. 2. Next POST /api/turns parses the Blob and calls buildCheckpoint again. boundFilesTouched now sees 256 paths → omitted=0. boundSummary sees .length > 8000, walks 8000 xs, hits the omitted-line newline, returns x×8000 + … [summary truncated]. 3. Seed user row has a lying truncation marker and no omitted-count. Same corruption for an at-cap emoji summary (🙂×8000 + omitted line). No test round-trips a pre-built checkpoint; the #954 row plants raw over-cap input (first-build only). |
Defender: the #954 plant test proves caps fire on hostile blobs; Phase 3 will call buildCheckpoint once at write; token trim is a different rail; renderSummaryRow still lists the 256 files. That fails: (a) this PR is the read seam Phase 3 will hit — write-once/read of an already-built checkpoint is the locked carrier; (b) persist is a dumb JSON.stringify(fold.compactionCheckpoint) with no builder; (c) omitted-count lives only on summary (shape has no omitted field) — dropping it is a Goal 4 honesty miss; (d) the truncation marker is a lie when the original head was at-cap and the extra bytes were the honesty suffix. |
high |
| Minor | L2 | Envelope PUT still bound-checks transcriptPointer and modelMessagesPointer (app/api/sessions/[id]/envelope/route.ts, adversarial #937: "seed still bind-checks, but a planted unbound id must not land on the envelope") and does not bound-check compactionPointer. validateMeta only requires isRedisSafeOpaqueId. A foreign Redis-safe id lands. Route boundSeedPointer refuses the read (no cross-session Blob read). |
1. Authed PUT /api/sessions/:id/envelope with meta.compactionPointer = newBlobObjectId(otherSession). 2. 200; pointer stored. 3. Next POST /api/turns treats it as unbound and falls through (modelMessagesPointer / sidecar / #937 503). No IDOR read, but a seed pointer of the same class as modelMessagesPointer now has a weaker PUT gate. |
Defender: host cloudMetaFor never emits the key; GET overlay does not fold it; only a raw API client can plant; the seed bind-check is the plan's confused-deputy DoD (row 5). True for the read. Still a #937-class hole on a seed pointer (freshnessReminderPointer is fail-open in-step, not this class). |
high |
| Minor | L6 | lib/agent/turnPersistSeam.test.ts gutted the locked #941 no-fold copy-forward proof. On main that test persist()s with fold: { cwd: 'docs' } and asserts freshnessReminderPointer survives. This PR closes the test after makeSeam seed and never calls persist() (seam unused). The persist() body was moved onto the new #949 no-compaction copy-forward row. |
A later overlay change that clears unspecified pointer keys: #949 compaction copy-forward still green; #941 freshness copy-forward is now a fixture-only assertion and would not catch the regression. | Defender: #941 production path is untouched; compaction has its own persist proof. True — this is a destroyed sibling proof, not a persist-seam logic bug. Still a vacuous test this PR introduced. | high |
Residual risk
Phase 3 (#950) is still the producer — this PR will not compact a live session until that lands, but the read path must already accept a well-formed buildCheckpoint Blob without corrupting it. A bound pointer whose Blob is missing still 503s only when the host also omitted promptHistory (locked #937 fail-closed, shared across both pointers). GET overlay / host sidecar-stop still keys solely on modelMessagesPointer; compaction is model-facing seed, not a paint/sidecar signal — correct for this phase. B7/B8 partial-commit (compaction Blob written, overlay lww_conflict) can orphan a checkpoint object the same way #936 / B6 already can; documented, not new. 503 copy still says "model-messages seed" when the miss was compactionPointer. cloudMetaFor's comment still lists only modelMessagesPointer / freshnessReminderPointer as the never-emit set (code is correct; comment is stale).
Merge guidance
CONCERNS: do not merge until the Major is fixed (buildCheckpoint must be a no-op on its own output — peel baked truncation/omitted suffixes, re-cap the head, re-apply omitted = filesOverflow ? n : peeledPrior, plus a round-trip test on a pre-built checkpoint). Minors should land in the same pass (PUT bound-check; restore the #941 persist() assertion).
What was not attacked
Live Workflows/'use step' serialization of a 1 MiB fold; prod Redis/Blob; DO runner; Gateway; Wasm paint; Phase 3 trigger/summarizer (#950); parent #947 close-out; npm run test:di-gate generated-file noise called out in the PR body.
…rsarial #954) Re-running buildCheckpoint on a persist Blob treated baked omitted/truncation suffixes as summary overflow: dropped the files-omitted honesty line and could stamp a lying truncation marker on an at-cap head. Peel those suffixes, recap the head, re-apply omitted = filesOverflow ? n : peeledPrior. Also: envelope PUT bound-checks compactionPointer (#937 class); restore the #941 no-fold persist copy-forward proof this branch had gutted.
|
Follow-up to the CONCERNS review on
Targeted vitest (364) + |
btipling
left a comment
There was a problem hiding this comment.
Adversarial review — PR #954
Verdict: CONCERNS
Repo: btipling/invincible
Scope: main ← plan/a4-compaction-p2 (fc0b8ea) · 18 files · A4 compaction phase 2 carrier + prefer-checkpoint seed (includes two follow-up commits that closed the prior #954 rounds)
Lenses run: L1, L2, L5, L6, L8 (skip: L3 no DOM/Wasm/chat surface; L4 no CI/artifact/workflow change — durable-turn int green; L7 no host/config bind; L9 no UI)
AGENTS.md read: yes (docs/feature-divide.md ownership of /api/turns; persist-step/API surface checked against SECURITY.md — no secret/runner reach)
Findings
| Sev | Lens | Finding | Break scenario | Refutation attempt | Confidence |
|---|---|---|---|---|---|
| Major | L1 | POST /api/turns builds [renderSummaryRow(...), ...retainedTail] then runs #944 trimModelMessagesToBudget over the combined seed. Trim drops oldest first. The honesty-labeled summary row is index 0, so it is the first casualty whenever summary+tail exceeds foldBudgetTokens. Route comment claims the dropped summary “is duplicated inside the checkpoint's baked-in overflow marker” — that marker is … (N earlier paths omitted) / … [summary truncated], not the summary text. Parent #947 Goal 4 (summary row is the seed) fails on the same budget the Phase-1 cut walk sizes the tail against. Path: app/api/turns/route.ts seed block; lib/agent/modelMessages.ts trimModelMessagesToBudget; test plan #949 row 2h locks drop-to-[] as intended. |
1. Bound checkpoint {summary: "auth model is X — do not revert", filesTouched: ["lib/auth.ts"], retainedTail: [fat last turn whose serialized size is just under the fold budget]}. 2. POST /api/turns on a 32k-class window (budget = window − 16384). Combined seed exceeds the rail. 3. Trim shifts start past index 0. Seed is the fat tail with no Summary of earlier session (compacted, not live assistant prose): row. 4. Model continues as if the tail is the full history; compacted span is gone and unlabeled. Reproduced in spirit by row 2h (800-token window → priorMessages: []). Same drop on any tail that fills leftover budget after the 8k-char summary (~2k tokens) is prepended. |
“Phase 3 cut walk leaves slack / only extreme windows miss” — findCompactionCut returns the newest fitting boundary (smallest tail that still fits), so a last-turn-near-budget is the intended cut, not an anomaly, and the summary is extra tokens the cut did not reserve. “Overflow marker duplicates the summary” — false; the marker is a cap-honesty suffix on checkpoint.summary, which lives inside the dropped row. “History must yield to the ask” — yes for the tail; Goal 4 is a labeled prefix, not oldest disposable context. Pin the summary (trim tail against leftover budget); yield to the ask only when the summary row itself cannot fit. |
high |
| Nit | L8 | Bound-pointer miss 503 still says “Unable to read the model-messages seed” when the only bound pointer was compactionPointer. Path: app/api/turns/route.ts shared #937 503; test row 2f asserts /model-messages seed/i. |
Operator/debug of a compaction-only miss reads as an #936 projection failure. |
Error is fail-closed either way; no user-facing harness copy. Rename to a pointer-agnostic “session seed”. | high |
Residual risk
Phase 2 is a write-once carrier: persist writes compactionPointer only when the fold carries a checkpoint; later no-fold persists copy-forward the frozen blob. Prefer-checkpoint then ignores live modelMessagesPointer. That is the parent lock (checkpoint = replacement seed; mm stays full-fidelity on disk). It will drop every turn after the compaction turn unless Phase 3 (#950) rewrites retainedTail on subsequent persists (or adds a watermark and merges). Do not treat the current seed as a live post-compact view across N uncompacted turns. Envelope GET overlayEnvelopeMeta still does not overlay compactionPointer (sidecar-stop stays on modelMessagesPointer); compaction-only persist therefore keeps sending promptHistory, which the route ignores on a successful checkpoint read and which saves the #937 503 on a miss — conservative, not a breach.
Prior rounds on this PR (46a8d48 planted-cap bypass, fc0b8ea buildCheckpoint idempotency + envelope PUT bind-check + restored no-fold copy-forward test) still hold at fc0b8ea. Persist still does not re-run buildCheckpoint on write; the route re-run is the belt.
Merge guidance
- CONCERNS: do not merge until the Major is fixed (pin the Goal 4 summary row across
#944trim; yield to the ask only when that row itself cannot fit). Nit optional.
What was not attacked
Live Vercel Blob/Redis, prod Gateway, DO runner, Phase 3 writer (derivePersistFold / summarizer step — not in this diff), Wasm/DOM paint, full vitest re-run in this workspace (CI durable-turn int is green on the head).
|
Follow-up to the CONCERNS review on
Targeted vitest (150: route + compaction + persist seam) + |
btipling
left a comment
There was a problem hiding this comment.
Adversarial review — PR #954
Verdict: CONCERNS
Repo: btipling/invincible
Scope: main ← plan/a4-compaction-p2 (9512bdb) · 18 files · A4 compaction phase 2 carrier + prefer-checkpoint seed (includes three follow-up commits that closed prior #954 rounds)
Lenses run: L1, L2, L5, L6, L8 (skip: L3 no DOM/Wasm/chat surface; L4 no CI/artifact/workflow change; L7 no host/config bind; L9 no UI)
AGENTS.md read: yes (docs/feature-divide.md ownership of /api/turns; persist-step/API surface checked against SECURITY.md — no secret/runner reach)
Findings
| Sev | Lens | Finding | Break scenario | Refutation attempt | Confidence |
|---|---|---|---|---|---|
| Major | L1 | COMPACTION_CHECKPOINT_MAX_BYTES = 1 MiB does not compose with the Phase-1 tail rail. findCompactionCut (lib/agent/compaction.ts) accepts a retained tail up to MODEL_MSG_SEED_MAX_BYTES (2 MiB) + MODEL_MSG_SEED_MAX_ROWS (4096). This PR's persist seam (lib/agent/turnPersistSeam.ts) JSON.stringifys that object and writeSegments under 1 MiB, fail-closed compaction_write_failed. The cap comment claims 1 MiB is "generous for … a bounded retained tail" — the bound is twice the cap. No producer in this PR sizes the object down; Phase 3 will call buildCheckpoint from a legal cut and this carrier will refuse it. |
1. A long turn whose newest user-boundary tail serializes to 1.2–2.0 MiB (many truncated tool-result rows; still ≤ the #944 seed byte rail, so Phase 1 returns a cut). 2. Phase 3 builds the checkpoint and passes it on fold.compactionCheckpoint. 3. Persist throws oversize → {ok:false, code:compaction_write_failed}; failWrite still overlays completed (+ any already-written mm/checkpoint pointers). 4. compactionPointer copy-forwards the old blob (or stays unset). Next POST /api/turns prefers the stale/missing checkpoint; this turn's compaction never becomes the seed. A 1.2 MiB tail is legal on main today (#948); this PR is the first writer and cannot carry it. |
"Newest-boundary cut is always tiny / 1 MiB is plenty" — the cut is the entire current turn (newest user row → end). A 512-step agent turn with batched tools at MODEL_MSG_TOOL_RESULT_MAX_CHARS (2000) routinely exceeds 1 MiB while still fitting 2 MiB. "Fail-closed is honest; Phase 3 will size to 1 MiB" — this PR is the write-once carrier; Phase 1 already defined the tail bound; a legal buildCheckpoint Blob that cannot persist is a carrier bug, not a Phase-3 note. "No production producer yet" — same argument used against the planted-cap / idempotency Majors on this PR; those were fixed on the read seam because Phase 3 will hit it. The write seam is the other half. |
high |
| Major | L1 | Goal 4 pin (commit 9512bdb) trims the retained tail with a token-only reservation: currentUserContent: prompt + 'x'.repeat(JSON.stringify([summaryRow]).length). trimModelMessagesToBudget applies three rails (lib/agent/modelMessages.ts): token (json.length + askChars), row (maxRows default 4096), byte (utf8Bytes(json) default 2 MiB Workflow-arg ceiling). Row/byte still see the tail alone. Then priorMessages = [summaryRow, ...trimmedTail] is passed to start() without a combined-rail check. Path: app/api/turns/route.ts seed block. |
1. Bound checkpoint whose tail fills the 2 MiB byte rail (legal Phase-1 cut once Finding 1's cap is raised, or a session-bound planted Blob — envelope PUT now accepts any bound id). 2. Summary row fits the leftover token budget (summaryFits true). 3. Tail trim keeps a ~2 MiB suffix (byte rail measured without the summary). 4. Combined seed is 2 MiB + summary serialization (label + 8k-char head + files line). start(turnWorkflow, { priorMessages }) exceeds MODEL_MSG_SEED_MAX_BYTES — the exact Workflow-arg overflow #944 exists to prevent. Same miss on rows: a 4096-row tail (cut's row rail) + summary = 4097. Pin test only asserts the honesty label survived, not utf8Bytes(JSON.stringify(priorMessages)) <= MODEL_MSG_SEED_MAX_BYTES. |
"1 MiB persist cap makes a 2 MiB tail unreachable" — that is Finding 1; fixing the cap without fixing the pin lights this up. "x.repeat over-reserves by a JSON bracket, so tokens are safe" — true for the token rail (~1 char conservative). Byte/row do not read askChars. "Host plant of an 8 MiB checkpoint-shaped transcript object is self-DoS" — authed POST /transcript + PUT envelope {compactionPointer} is a first-party API client, same class #937 already bound-checked; the mm seed path trims the combined array and would not overflow. |
high |
Residual risk
Prior rounds on this PR still hold at 9512bdb: planted-cap re-run of buildCheckpoint, peel/idempotency of honesty suffixes, envelope PUT bind-check, restored #941 no-fold persist proof, Goal 4 token pin (incomplete — see Major). Phase 3 (#950) is still the producer — this PR will not compact a live session until that lands. Prefer-checkpoint ignores live modelMessagesPointer; post-compact turns after the cut are dropped until Phase 3 rewrites retainedTail (or adds a watermark). GET overlayEnvelopeMeta still does not overlay compactionPointer (sidecar-stop stays on modelMessagesPointer) — conservative. B7/B8 partial-commit (compaction Blob written, overlay lww_conflict) can orphan a checkpoint object the same way #936 / B6 already can. failWrite after a successful mm write + failed compaction write will overlay the new modelMessagesPointer while copy-forwarding the old compactionPointer — next seed prefers the stale checkpoint; documented sibling partial-commit, worse here because the preferred pointer is the one that did not advance. Living docs (SECURITY.md blob-contents list, feature-divide seed paragraph) still name only modelMessagesPointer; plan locked docs to parent Phase 4 (#951).
Merge guidance
- CONCERNS: do not merge until both Majors are fixed together. Raising the persist cap without measuring the combined seed on all three
#944rails re-opens the Workflow-arg overflow. Pin the summary by measuring[summaryRow, ...tailSlice]on token and byte and row (apinnedCount/ equivalent ontrimModelMessagesToBudget), and setCOMPACTION_CHECKPOINT_MAX_BYTESto compose withMODEL_MSG_SEED_MAX_BYTESplus envelope slack for summary/files/keys — still well under the 8 MiB mm object and the 4.5 MB Function wire. This is a NEW unmerged cap (not an existing-on-main value); the raise is the composition fix, not a human-gate change to a shipped ceiling. - Tests: combined-seed byte/row assertions on the pin path; persist cap relationship locked next to
MODEL_MSG_SEED_MAX_BYTES.
What was not attacked
Live Vercel Blob/Redis, prod Gateway, DO runner, Phase 3 writer (derivePersistFold / summarizer step — not in this diff), Wasm/DOM paint, 'use step' serialization of a 2 MiB fold, full vitest re-run in this workspace (CI durable-turn int last reported green on the head).
…m (adversarial #954) Goal 4 pin measured only the token rail via fake currentUserContent padding, so row/byte rails could overflow MODEL_MSG_SEED_* on the combined seed. pinnedCount now trims the suffix while every rail sees [summary, ...tailSlice]. COMPACTION_CHECKPOINT_MAX_BYTES raised from 1 MiB to MODEL_MSG_SEED_MAX_BYTES + 256 KiB slack so a Phase-1-legal findCompactionCut tail can persist (NEW unmerged cap; no existing-on-main ceiling changed).
|
Follow-up to the CONCERNS review on
Targeted vitest (175: route + compaction + persist seam + modelMessages) + |
btipling
left a comment
There was a problem hiding this comment.
Adversarial review — PR #954
Verdict: PASS WITH NOTES
Repo: btipling/invincible
Scope: main ← plan/a4-compaction-p2 (186a3c9) · 20 files · A4 compaction phase 2 carrier + prefer-checkpoint seed (includes four follow-up commits that closed prior #954 rounds)
Lenses run: L1, L2, L5, L6, L8 (skip: L3 no DOM/Wasm/chat surface; L4 no CI/artifact/workflow change; L7 no host/config bind; L9 no UI)
AGENTS.md read: yes (docs/feature-divide.md ownership of /api/turns; persist-step/API surface checked against SECURITY.md — no secret/runner reach)
Findings
| Sev | Lens | Finding | Break scenario | Refutation attempt | Confidence |
|---|---|---|---|---|---|
| Nit | L8 | Plan issue #949 Caps table still lists NEW COMPACTION_CHECKPOINT_MAX_BYTES = 1 MiB. Code (and the 1.2 MiB persist test) is MODEL_MSG_SEED_MAX_BYTES + 256 KiB. Path: issue #949 vs lib/sessionCloudCaps.ts COMPACTION_CHECKPOINT_MAX_BYTES. |
Phase 3 (#950) implement-plan reads the issue caps table, sizes derivePersistFold / a producer-side write to 1 MiB, and fail-closes a Phase-1-legal ~1.2–2.0 MiB tail — re-introducing the Major this commit closed. |
“Phase 3 will import the constant, not copy the literal.” True if the implementer reads sessionCloudCaps.ts. The issue is the phase’s locked caps table; every other A4 phase has treated that table as the producer brief. Cheap to keep it honest. |
high |
Residual risk
Phase 3 (#950) is still the producer — this PR will not compact a live session. Prefer-checkpoint then ignores live modelMessagesPointer until the next compaction rewrite of retainedTail (parent lock: checkpoint = replacement seed; mm stays full-fidelity on disk). Do not treat the current seed as a live post-compact view across N uncompacted turns.
GET overlayEnvelopeMeta still does not overlay compactionPointer (sidecar-stop stays on modelMessagesPointer); compaction-only persist therefore keeps sending promptHistory, which the route ignores on a successful checkpoint read and which saves the #937 503 on a miss — conservative, not a breach. B7/B8 partial-commit (compaction Blob written, overlay lww_conflict) can orphan a checkpoint object the same way #936 / B6 already can.
A maxed Phase-1 tail (row/byte/token rails full) plus the Goal 4 summary row (and the new ask) can exceed the seed rails; pinnedCount: 1 keeps the honesty row and drops oldest tail. That is seed-rail truth, not a Goal 4 miss. Phase 3 may want the cut walk to reserve summary tokens/rows/bytes so a maxed tail is not trimmed again on seed.
256 KiB checkpoint slack covers summary + keys + short workspace paths. A 256 × PATH_MAX filesTouched list plus a 2 MiB tail will compaction_write_failed (fail-closed, no truncation lie). Phase 3 should bound path lengths (freshness reminder already has a byte rail).
persistStep fold will carry modelMessages (≤8 MiB, pre-existing) plus compactionCheckpoint (≤2.25 MiB) as 'use step' args once Phase 3 wires the producer. Confirm Workflows serialization headroom there; the Blob object itself is not a Function body.
Prior rounds on this PR (46a8d48 planted-cap bypass, fc0b8ea buildCheckpoint idempotency + envelope PUT bind-check + restored no-fold copy-forward, 9512bdb Goal 4 pin + 503 copy, 186a3c9 cap composition + combined-rail pinnedCount) still hold at 186a3c9. Persist remains a dumb JSON.stringify(fold.compactionCheckpoint); the route re-run of buildCheckpoint is the belt.
Merge guidance
- PASS WITH NOTES: safe to merge from this attack; the Nit is optional (update #949 caps table to
MODEL_MSG_SEED_MAX_BYTES + 256 KiBso Phase 3 does not re-litigate the 1 MiB ceiling). - Do not treat “Phase 3 will call
buildCheckpoint” as a substitute for the read-side belt — that is already in the route.
What was not attacked
Live Vercel Blob/Redis, prod Gateway, DO runner, Phase 3 writer (derivePersistFold / summarizer step — not in this diff), Wasm/DOM paint, Workflows 'use step' serialization of a 2.25 MiB fold next to an 8 MiB modelMessages sibling, full vitest re-run in this workspace.
|
Follow-up to the PASS WITH NOTES review on
No PR code change — HEAD |
Plan #949 — A4 compaction phase 2: checkpoint carrier + prefer-checkpoint seed
Implements the HANDOFF-READY plan in #949 (phase 2 of parent #947, source #552). Phase 1 landed in #953.
What this lands
The write-once/read carrier for the compaction checkpoint — no trigger, no summarizer (Phase 3, #950):
COMPACTION_CHECKPOINT_MAX_BYTES= 1 MiB (lib/sessionCloudCaps.ts) — generous by default; no existing cap value changed → no human gatecompactionPointer(4th worker-authored sibling aftermodelMessagesPointer/workingNotes/freshnessReminderPointer):RESERVED_META_KEYS+copyForwardCompactionPointer+ drop-to-unset sanitize arm inlib/sessions/sessionStore.ts; wired into bothupsertEnvelopeimplementations (Memory + Redis)compactionPointeradded toWORKER_META_KEYS+sanitizeWorkerKeyValuearm (lib/agent/workerMetaOverlay.ts) — DISTINCT from the older B6checkpointPointersiblingcompactionCheckpoint, the seam writes the typed{summary, filesTouched, retainedTail}as its OWN session-bound Blob object underCOMPACTION_CHECKPOINT_MAX_BYTES(writeSegment), fail-closedcompaction_write_failed, and setspatch.compactionPointer. No fold → no write; the prior pointer survives via B8 copy-forward (durable — no #941-style volatility)POST /api/turnsprefers the checkpoint: re-validates the shape, re-pairs the retained tail viabuildModelMessages, seeds[renderSummaryRow(...), ...retainedTail](honesty-labeledusersummary row FIRST — never live assistant prose), then falls backmodelMessagesPointer→ legacypromptHistory(locked fallback chain). Same DI surface (services.createBlobTranscriptStore()— no new seam); the #937 fail-closed 503 now covers both bound pointers; the #944 token trim runs over the combined seedcloudMetaFor/trimForCloudPutnever emitcompactionPointer(worker-authored; envelope PUT copy-forwards on omit — a host flatten can never clear it)Layers
Tests (26 new; full suite 176 files / 3450 tests green)
lib/agent/turnPersistSeam.test.ts— rows 1a/1b/1c: own-Blob write + pointer-in-meta-only, fail-closedcompaction_write_failed, no-fold copy-forward durability, pointer advance on a new compactionlib/sessions/sessionStore.test.ts— reserved-key list, accept/drop-to-unset round-trip,copyForwardCompactionPointer(explicit wins / poison not copied),upsertEnvelopecopy-forwardlib/agent/workerMetaOverlay.test.ts— compactionPointer sanitize; B6 sibling preservedapp/api/turns/route.test.ts— rows 2a–2h: prefer-checkpoint seed with re-pair, malformed → fallback, null-read → fallback, no-cp → plan: structured truncated tool_result on the wire (A1, source #549) #936 path, unbound → legacy fold, bound-miss-no-sidecar → 503, both absent → legacy sidecar, plan: context-window fold budget (A3, source #551) #944 trim over the checkpoint seedlib/sessionRepository.test.ts— hostcloudMetaFornever emits the keyVerification
npm run typecheck✅vitest run --project default --project tenancy: 176 files / 3450 tests — all passnpm run test:di-gatefails identically on a pristine checkout in this workspace — the git-ignored generatedapp/.well-known/workflow/v1/step/route.js(Workflow bundler output) trips the static scan. CI (int-durable) does not run di-gate and is green on this branch's Phase-1 sibling runs; the diff here touches no DI surface.Closes #949. Refs #947 (parent), #552 (source A4), #948/#953 (phase 1).