agent session architecture — model-context memory, compaction, and viewport

[btipling]

Title intent

This is the project-tracking / roadmap issue for agent session intelligence
and viewport. It sequences work that #408 (session object: id, cwd, sandbox,
Redis, Blob) deliberately did not cover: what the model actually sees,
how the canvas pages history, and how first paint stays cheap.

#408 is the durable addressable session. This issue is the durable
agent mind + scroll window on top of that object.

Estimates and phase boundaries are directional. Each phase must be planned as
its own parent create-plan issue before implementation.

Status (2026-09-05): OPEN (umbrella). Diagnosis is locked. Shipped:
A1 #549 (plan #936 / PR #937,
Buffer follow-up PR #939),
A1-adj #693 (plan #941 / PR #943),
A2 #550 (plan #938 / PR #940),
A3 #551 (plan #944 / PR #945),
A6 #557 (plan #931 / PR #932).
Next on this board: A4 #552. #694 stays Later. Still do not
open a fifth identity issue.

Goal

Stop treating the session as an app payload (400 rows, 3.5M char slice,
8 MiB Blob, 8 MiB localStorage, paint everything). Treat it as:

1. Agent intelligence — the model should remember what it did and what it
   learned across turns, up to the model’s real context window (hundreds
   of thousands to ~1M tokens on current catalog models). Asking “what did you
   already find?” should work.
2. A viewport, not a dump — the canvas scrolls a window. Almost never
   the whole 8 MiB. First paint must not JSON.stringify megabytes onto
   localStorage (Stop freeze).
3. One Blob still OK — keep a single transcript object if we can; spike
   turn-seam byte offsets (or equivalent) so we can page without sharding
   into thousands of part files.

The operator should not need to scroll at all if the agent still knows the work.

The lobotomy (2026-08 diagnosis)

Production durable turns no longer match this table for inference: A1
shipped a structured model-messages projection (meta.modelMessagesPointer);
A1-adj shipped a volatile per-turn freshness reminder (meta.freshnessReminderPointer);
A2 shipped session-owned meta.workingNotes; A3 shipped a window-derived
token fold budget; A6 shipped catalog-only skill inject. Remaining first-turn /
warehouse pain is A4 (no compaction) and A5 (viewport / slim local).
The rows below are the 2026-08 baseline.

Layer	What we do today	Why it feels broken
Persist	Blob / local snapshot up to 8 MiB, including dense tool_run cards	Stop persist + wasm_refresh hitch; first paint overpays
Inference	Single { prompt } string via formatPromptWithHistory — last 400 user/assistant/system/error rows, then 3.5M character tail. No tool_use/tool_result messages	Capped like an HTTP body; next turn cannot see tools
Tool evidence	tool_run is display-only — never folded	Agent re-reads files every turn; no session memory of tool work
Thinking	Ephemeral, not in SessionStore	Fine. But nothing else carries findings → each turn is a new assistant
Canvas	Hydrate up to 2048 ring rows; Load earlier 512	Paints work that is rarely revisited
Cloud	One JSON { messages: [...] } Blob	Cannot Range into “latest page” of a JSON array

Symptom: the assistant refers to “the previous assistant,” forgets research,
and the tab freezes on Stop — orthogonal problems, same root: we store a
warehouse and ship crumbs to the Gateway.

See Identity dissociation (2026-08-18) below for the four fault lines
and the streamText response.messages lock on A1.

Target (directional — lock in phase plans)

Session (already shipped, #408)     = id, tenant, cwd, sandbox, pointer
AgentMemory (this umbrella)         = working notes + folded tool evidence
                                       + compacted history sized to model window
Viewport (this umbrella)            = slim local tail + Blob pages/offsets
                                       + scrollbar that feels infinite

Key invariant: inference budget ≠ storage budget ≠ paint budget.

• Storage may keep the full Blob (8 MiB object ceiling, not a Function body).
• Paint may keep a ring window (2048 is a canvas cap, not an intelligence cap).
• Inference uses model context tokens (catalog window − reserve; A3 inference budget: replace 400-row / 3.5M-char fold with model-window token budget #551 shipped). Occupancy metering is status bar: show current context occupancy in addition to summed turn usage #556. Compaction when over budget is compaction: model-capability context compaction (not API payload slices) #552. 400 / 3.5M are not intelligence caps.

Source issues

Source	Title	Track	Status
#548	Agent session architecture (parent)	roadmap	⬜ OPEN
#549	Structured messages — truncated tool_result on the wire (not Tool: crumbs)	intelligence	✅ CLOSED — PR #937 (plan #936); follow-up PR #939
#693	Tool-result snapshots ≠ live files (re-read or they’re stale)	intelligence	✅ CLOSED — PR #943 (plan #941)
#550	Durable working notes / identity across turns	intelligence	✅ CLOSED — PR #940 (plan #938)
#551	Replace 400 / 3.5M fold with model-window budget	intelligence	✅ CLOSED — PR #945 (plan #944)
#552	Model-capability compaction engine	intelligence	⬜ OPEN
#553	Spike: Blob viewport + intelligent scrollbar + slim local	viewport	⬜ OPEN — spike
#556	Context occupancy (last/max step) + keep turn totals	meter	⬜ OPEN — not this board (status bar). #327 closed (totals only)
#557	Skills: inject catalog, not full bodies	intelligence	✅ CLOSED — PR #932 (plan #931)
#558	Cache-stable two-block system prompt	intelligence	⬜ OPEN
#536	Session TODO lists	adjacent	⬜ OPEN — phase tags feed #552 (not this board unless we make the loop phase-driven)
#560	Per-turn fold records (measure #549/#558)	measure	⬜ OPEN
#372	Agent templates / forking (save good context)	templates	⬜ OPEN — blocked by this umbrella (moved off #408 / project 2). A1+A3 shipped; still needs A4 so a snapshot is honest compacted memory, not a raw window dump
#694	Cross-session handbook (idle extract, default off)	later	⬜ OPEN — not the first-turn fix; blocked by A4 (A1+A3 shipped)

Baseline (do not re-litigate): #408 core session object, envelope + Blob
carrier (#515), tool_run display-only (#345), thinking collapse, status bar
#328 / #538. File mutate freshness (#277 RunFileFreshness) is shipped and
stays per-POST — do not put fingerprints on the session (#693).

Sequence / phases

Intelligence first (forgetting is the product bug). Viewport can spike in
parallel once slim-local is specified; do not block memory work on pagination.

Phase	Issue	What ships	Status
A0	#548	Diagnosis locked (this issue)	✅ locked (umbrella stays OPEN)
A1	#549	Structured tool results on the wire (truncated tool_use/tool_result)	✅ PR #937
A1-adj	#693	Snapshots ≠ live files — empty-ledger reminder so A1 composes with #277	✅ PR #943 (plan #941)
A2	#550	Working notes / identity	✅ PR #940
A3	#551	Model-window budget (drop 400 / 3.5M as intelligence caps)	✅ PR #945 (plan #944)
A4	#552	Compaction engine (checkpoint + retained tail)	⬜
A5	#553	Viewport spike (one Blob + offsets or pages; scrollbar; slim local)	⬜ spike
A6	#557	Skill catalog inject (bodies on fetch/attach only)	✅ PR #932
A7	#558	Two-block cache-stable system	⬜
A8	#560	Per-turn fold records (no router)	⬜
Later	#694	Cross-session handbook, default off	⬜ after A4

A3 shipped separately from A4 (budget is the policy; compaction is the
mechanism). A4 is next.

A1 lock (2026-08-15): #549 is native / structured tool_result on the
Gateway payload, not more prose in formatPromptWithHistory.

A1-adj lock (2026-08-19): #693 is not a fifth identity issue. A1 has
shipped, so the model does see last turn’s read_file and can think it has
an edit grant; #277’s ledger is empty every Function POST. Reminder plan is
#941 (source #693).
Shipped 2026-09-05: PR #943
(merge b0a81d6). Advisory volatile reminder; #277 stays the fail-closed gate.

A3 lock (2026-09-05): #551 is a window-derived token budget, not a tokenizer
and not occupancy. Durable seed trim at the /api/turns route boundary
(trimModelMessagesToBudget); host legacy fold (formatPromptWithHistory) is
token-budget first. Reserve is Pi-style max(16384, 15% × window); tokens are
ceil(chars/4). Plan #944 /
PR #945 (merge 9a049c6).
Occupancy meter stays #556; compaction stays #552.

Peer harness map (OpenCode / Pi / Oh My Pi / Codex / DeepSeek)

Steal payload ideas. Skip their disk/process model (JSONL, SQLite,
part/*.json, Hashline, LSP-on-write, subagent worktrees, mailboxes,
homedir ~/.codex/memories).

Peer idea	Ours
Messages array with tool_use / tool_result	#549 (A1)
Truncate tool output (~2k–50KB), don’t omit	#549
Spill oversized bytes; locator in the result (DeepSeek / OMP artifact://); fail if retain fails (OpenCode)	#549 + #565 / #563
Reload projected history every provider turn (OpenCode)	#549 — we already reload (Functions); persist response.messages, stop flattening paint
Persist compaction as the next request (Codex replacement_history; OpenCode typed compaction message)	#552
CustomMessageEntry / durable notes	#550
Compact at window − reserve (~16k)	#551 + #552
retainedTail + files-touched in the summary	#552
Cursor / offset pagination	#553 (spike; one Blob preferred)
Skills XML/catalog; bodies on fetch	#557
Two-part system for KV cache	#558
Context Epoch / world-state diffs (freeze baseline; chronological system diffs; rewrite prefix only at compact)	#558
Snapshot a good window as a template	#372 (after A1–A4)
Last-step occupancy vs turn-sum usage	#556 — not this board
Durable-spec slot (compact into stable notes)	#550 + #552
Compact / family-switch only at phase boundaries	#552 (phase from #536)
Stay-put default; Next = cache event	#551 / #558
Choice precedes assembly	#558
Tool/MCP list snapshot; refresh at phase boundary	#557
Exec: summary in context, verbose on disk	#549
Per-turn fold record (no router)	#560
Codex CLI: append-only Responses input; inner cycle = reasoning + function_call + function_call_output before yielding	#549 (same invariant; we use streamText messages, not Responses items)
DeepSeek: replay reasoning_content + tool_calls verbatim or attention collapses	#549 A1 per-provider lock (see Identity dissociation)
streamText response.messages = full turn graph (not terminal assistant string)	#549 A1 mechanism
Snapshots in the transcript, freshness beside it (DeepSeek observation; OMP hashline; we already have #277 per-POST)	#693
Two memory systems: live thread ≠ deferred handbook (Codex / OMP; default off like OMP)	#694 Later

Identity dissociation — operator report + research (2026-08-18)

Operator: agents do not recognize the previous session as themselves.
Each new send feels like the model’s first turn. This is not a foundation-model
defect.

It is the same “previous assistant / new hire each send” symptom in the
lobotomy table. A 2026-08-18 review of Pi, Oh My Pi, OpenCode, Codex CLI,
DeepSeek, and Vercel AI SDK streamText maps it onto four harness fault
lines we already have children for. Do not open a fifth issue.
A1 (#549) has since shipped (PRs #937 / #939); A1-adj (#693 / plan #941 /
PR #943) shipped 2026-09-05.
A3 (#551 / plan #944 / PR #945)
shipped 2026-09-05. Remaining first-turn / warehouse pain is A4 (compaction) and A5 (viewport).

Fault line	What we do today	Child
1. Intermediate-step pruning	Shipped (A1). Durable turns persist truncated tool_use/tool_result via meta.modelMessagesPointer. 2026-08 baseline: formatPromptWithHistory folded only user/assistant/system/error into one { prompt } string; tool_run was display-only (#345).	#549 A1 ✅
2. Reasoning / providerMetadata stripped	Paint thinking stays ephemeral (correct). Model-facing replay is an A1 lock (per-provider). 2026-08 baseline dropped reasoning parts and Anthropic/DeepSeek signatures.	#549 A1 ✅ (model-facing only). SessionStore CoT stays out (non-goal).
3. Role inversion	Shipped (A1). Structured toolCallId pairing on the wire. 2026-08 baseline: legacy system tool lines became Tool: … prose with no pairing.	#549 A1 ✅
4. Non-atomic tail slice	Shipped (A3). Fold budget is catalog context_window − Pi-style reserve (max(16384, 15% × window)), tokens ceil(chars/4). 400 is a row-count safety rail; 3.5M is a host transport backstop. Compaction (when the window is full) is still A4. 2026-08 baseline: dialogue.slice(-400) then out.slice(-3_500_000).	#551 A3 ✅ + #552 A4

#550 A2 (working notes) is complementary, not the first-turn fix. A notes
block on top of a pruned transcript is still a new assistant reading someone
else’s diary. A1 shipped first (evidence to remember); A2 notes now carry
identity until A4 compaction exists to write into.

A1 mechanism — streamText response.messages (shipped)

lib/agent/runAgent.ts uses streamText + stepCountIs. The SDK’s
onFinish / result.response.messages is the complete ModelMessage[]
for the turn: assistant tool-calls, role: 'tool' results, reasoning parts,
final text — not the terminal assistant string the 2026-08 fold flattened.

A1 persists that array (bounded / truncated) as the model-facing history
and sends it as a messages array on turn N+1. Do not add more Tool:
lines to formatPromptWithHistory. Paint stays tool_run cards
(feature-divide). Abort/cancel: persist committed tool pairs the same way we
persist change_dir.

Peer split-result (Pi): LLM payload ≠ UI payload. We already have that split
in spirit (tool_run paint vs what A1 must send). Do not stuff paint JSON
into the Gateway.

Reasoning lock (do not silently flip)

• Paint / SessionStore: thinking rows stay ephemeral. Do not dump CoT
  into Redis/Blob as the memory mechanism.
• Model-facing replay: A1 locked, per provider, whether reasoning
  parts + providerMetadata ride the structured messages array (required
  for DeepSeek-R1 / Anthropic thinking continuation) or stay dropped. That
  remains an A1 lock, not a new issue.

Compaction (A4) — tool-pair atomicity

When we compact, an assistant tool-call and its tool-result are kept or
summarized together. Never slice(-N) across that boundary. Checkpoint
is labeled summary, never live assistant prose (#552 already).

What this research does not change

• session architecture — addressable per-user sessions with session-owned state (cwd, sandbox binding) #408 session object / envelope / Blob
• Dual chat / Wasm talking to Redis or Blob
• Pi JSONL DAG, OMP loopback workers, or Codex /responses/compact as our
  runtime — steal the invariant, not the process
• Implementing any phase in this issue

Orrery takeaways (2026-08-15)

Steal when and measure. Do not steal their runtime.

• Compaction is a cache flush — phase boundary or hard ceiling only; write into a durable-spec slot, not a churning history head.
• Stay-put is a decision. Operator Next mid-session is priced as a prefix re-read.
• feat: session TODO lists — one item per turn, host auto-continues with full list in the prompt #536 phase tags are how compaction: model-capability context compaction (not API payload slices) #552 knows the boundary. Keep feat: session TODO lists — one item per turn, host auto-continues with full list in the prompt #536 off this board until the loop is phase-driven.
• A fold record (telemetry: per-turn fold records (know if cache/memory work landed) #560) is how we know A1/A6/A7 worked. Policy/eval/hashline/spawn stay out.

Skip: Go binary, SQLite SoT, hashline, first-party multi-provider router, localhost-no-auth, learned policy, replay eval.

Source-level peer notes (2026-08-19)

Four investigations of non-cloud harnesses (Codex, DeepSeek Harness, OpenCode,
Oh My Pi). Comment on this issue has the ranked steal/skip list. Locks that
belong on children, not a new identity bug:

1. Two systems. Live thread (agent memory: structured truncated tool_result on the wire (not Tool: crumbs) #549–compaction: model-capability context compaction (not API payload slices) #552, agent memory: durable working notes / findings across turns (identity, not one-shot) #550) ≠ deferred handbook (later: cross-session handbook (idle extract, default off) — not the first-turn fix #694).
   Mixing them is how compaction wipes a cat while the handbook never had it.
2. Persist ≠ fresh. After A1, tool_result is a snapshot. plan: agent read-before-edit + external file freshness (parent) #277 already
   empties the observation ledger every Function POST. Prompt-side reminder is
   agent memory: tool_result snapshots are not live file views (re-read or they’re stale) #693. Do not rewrite results; do not put fingerprints on meta.
3. Reload a durable array. OpenCode re-queries SQLite every llm.stream.
   We already reload (stateless Functions). A1 persists response.messages
   and stopped flattening tool_run paint (agent memory: structured truncated tool_result on the wire (not Tool: crumbs) #549).
4. Persist the compact replacement. Codex Compacted.replacement_history /
   OpenCode typed compaction message so resume == live post-compact view
   (compaction: model-capability context compaction (not API payload slices) #552). We have no FTS over shadowed history — files-touched + notes must
   carry the investigation.
5. Diff the world. Codex world-state / OpenCode Context Epoch: freeze block 1;
   chronological system diffs; rewrite the prefix only at compact (inference: cache-stable two-block system prompt (don’t bust the KV prefix) #558).
   OMP sticky RULES.md: re-attach standing orders near the leaf so A4 cannot
   bury the persona snapshot.
6. Don’t write the handbook on the hot path. Idle extract, no-op is valid,
   default off (OMP, not Codex-on-by-default). Cloud-shaped, not ~/.codex.
   later: cross-session handbook (idle extract, default off) — not the first-turn fix #694 Later.

Skip (unchanged): homedir JSONL, client SQLite, snapcompact PNGs, hashline as
the edit dialect, mnemopi embeddings as core, file-watcher inbox as v1.

Non-goals (this umbrella)

• Replacing Redis / envelope / server-minted ids (session architecture — addressable per-user sessions with session-owned state (cwd, sandbox binding) #408 stays)
• CRDT / real-time co-edit
• OpenCode’s thousands of part/*.json files or client SQLite as SoT
• Putting raw thinking blobs into SessionStore by default
• Sending 8 MiB through a Vercel Function (4.5 MB ceiling; 2 MiB function-body cap stays)
• Dual DOM chat / Wasm talking to Blob
• Implementing any phase in this issue
• Hashline, in-process bash, mid-stream token abort, subagent worktrees
• Provider anthropic.txt / beast.txt packs (optional later, not this board)
• Default-on cross-session handbook / live MEMORY.md writes during runAgent (later: cross-session handbook (idle extract, default off) — not the first-turn fix #694 is later, off)
• Persisting plan: agent read-before-edit + external file freshness (parent) #277 file fingerprints on the session blob (agent memory: tool_result snapshots are not live file views (re-read or they’re stale) #693)

Board

Only agent session architecture (user project 3).

The agent-session label is the auto-add trigger for this board. Do not also put #548–#558 / #560 / #372 / #693 / #694 on the #408 session-architecture board (project 2). That board is the shipped session object track. This umbrella is a different product problem (model memory + viewport).

Phase issues

• agent memory: structured truncated tool_result on the wire (not Tool: crumbs) #549 A1 — structured truncated tool results on the wire (PR #937)
• agent memory: tool_result snapshots are not live file views (re-read or they’re stale) #693 A1-adj — snapshots ≠ live files (PR #943; plan plan: volatile per-turn freshness reminder — snapshots are not live file views (A1-adj, source #693) #941)
• agent memory: durable working notes / findings across turns (identity, not one-shot) #550 A2 — working notes / identity (PR #940)
• inference budget: replace 400-row / 3.5M-char fold with model-window token budget #551 A3 — model-window budget (PR #945; plan plan: context-window fold budget (A3, source #551) #944)
• compaction: model-capability context compaction (not API payload slices) #552 A4 — compaction engine
• spike: Blob viewport — turn-seam offsets, intelligent scrollbar, slim first-paint local #553 A5 — viewport spike
• skills: inject a catalog, not full bodies, every turn #557 A6 — skill catalog (not full bodies) (PR #932)
• inference: cache-stable two-block system prompt (don’t bust the KV prefix) #558 A7 — cache-stable system blocks
• telemetry: per-turn fold records (know if cache/memory work landed) #560 A8 — fold records
• feat: agent templates / forking (save good context, reuse later) #372 templates / fork — after A4 (A1+A3 shipped; still blocked by compaction)
• later: cross-session handbook (idle extract, default off) — not the first-turn fix #694 Later — cross-session handbook (default off; blocked by A4)