[RFC]: Reuse shared observations across decision questions — community help wanted

[hsliuustc0106]

Motivation

Agent workloads often ask several questions about the same observation. Independent full prefills can repeatedly compute an expensive image/video or text prefix.

Valen issue #29 documents a concrete example: its Qwen path shares processor encoding but repeats backbone execution per question and per Score level. Fifteen five-level Score questions require 75 explicit backbone calls. This is a source-level call count, not a measured speedup opportunity.

System1-Omni could make shared-observation serving useful across compatible decision models.

Proposed first step

Produce a focused design and benchmark plan before implementing a general cache or scheduler. Select one runnable model and freeze paired request manifests with one observation and increasing numbers of questions/candidates.

Hypothesis: reusing compatible observation computation reduces total multi-question latency while preserving reference decisions and declared numerical tolerances. Memory cost and cache-hit/miss behavior must be measured alongside latency.

Questions for community feedback

1. Which reuse boundary is both correct and valuable first: preprocessing, vision features, or the language prefix's execution state?
2. For Qwen's hybrid attention/recurrent layers, what state must be retained or copied to execute independent branches correctly?
3. What defines compatibility and invalidation: model/adapter revision, media/preprocessing identity, prompt prefix, positions and precision?
4. How should memory be bounded and state ownership handled across requests?
5. Would candidate/question batching be a simpler first improvement for the selected workload?

Preserve independent question semantics. In particular, Valen Score branches see only their own level description; concatenating all levels/questions into a new prompt changes the computation.

RFC deliverables

• One concrete workload and pinned reference, plus a small reproducible request manifest.
• Current call counts, a latency breakdown where measurable, and the chosen reuse boundary with alternatives and tradeoffs.
• A correctness plan covering ordering, different observations/candidates, invalidation and declared numerical tolerances.
• A comparison plan with frozen controls, a run budget, success/stop criteria, memory measurements and raw results.
• An explicit API/state-ownership proposal, or a statement that the first slice leaves the API unchanged.
• Small follow-up implementation tasks after feedback; report negative or inconclusive results as well.

Use the existing benchmark protocol and architecture contracts. Valen measurements depend on its reference-worker integration (#84); an already runnable text model can support an initial study.

Community help wanted

We welcome real multi-question agent traces, workload design, CPU request fixtures, profiling, and review of cache/recurrent-state correctness. Please comment with a workload or a boundary you can investigate.

Feedback is open; no implementation deadline or speedup is committed by this RFC.

[twu3202]

On question 2, for the qwen3_5 CUDA backend: each Gated DeltaNet layer needs its recurrent state after the prefix (float32, as gdn_prefill.cu keeps it) and the last three convolution inputs per channel; each full-attention layer needs the prefix keys (after norm and RoPE) and values; text positions continue from the prefix length. Since a branch never runs past its own end, kernels that read the saved state without writing it back let one copy per prefix serve every branch:

	Per prefix: GDN state + conv inputs	Per prefix token: KV
Cua-S1, Qwen3.5-4B	49 MiB	32 KiB
Open-Jev, Qwen3.8-27B	147 MiB	64 KiB

None of this is exposed yet: cs1_gdn_prefill has no initial or final state, cs1_gdn_conv no history, and cs1_attention one length for queries and keys.

For a first workload, Open-Jev fits best: every candidate prompt starts with Context:\n{state}\n\nQuestion: .... Cua-S1 puts the goal before the state, so its questions share little. On question 5, each candidate is a separate batch-1 forward today, so for short states, running a question's branches in one call may matter more than the reuse; the two combine.

If that direction looks right, I can write the plan for Open-Jev first, then send the backend changes as small PRs tested against a full forward. I can run the Open-Jev path end to end with Open-Jev-9B, the same method on Qwen3.5-9B.

[hsliuustc0106]

Thanks @twu3202. Yes, Open-Jev looks like a good first workload. Please post the plan here first, then split the backend changes into small PRs with the existing full forward retained as the reference.

For the first slice, I’d keep prefix reuse within one request and the HTTP API unchanged. A few points to cover in the plan:

• Share an immutable prefix snapshot, with branch-local mutable state/workspace. Find the common prefix from the actual token IDs, including the chat template, and preserve positions and causal masking.
• Check convolution history and GDN continuation at short prefixes and around the 64-token chunk boundary, including splits inside a chunk. Splitting the computation can change rounding, so declare numerical tolerances and check hidden states, head outputs and final decisions against the full forward.
• Compare independent forwards with prefix reuse first. Candidate batching is useful too, but measure it separately so we can tell which change helps for short versus long contexts.
• Include the memory calculation from model dimensions, and distinguish retained prefix state from per-branch scratch. Keep branch order and repeated-use checks in the correctness cases.

Open-Jev-9B is reasonable for initial feasibility and parity work. One practical hurdle: the current head loader explicitly requires the 27B backbone dimensions. Please identify any 9B loading/export changes as a separate preparation step and validate them against its own pinned reference, head and temperature. Record results as 9B measurements; any 27B performance claim needs a corresponding 27B run.

[hsliuustc0106]

@twu3202, please also base the plan and follow-up PRs on the updated architecture and integration contracts and native runtime boundaries.

For prefix reuse, the ownership should be explicit:

• The shared Qwen executor owns retained prefix/recurrent/KV state, branch scratch, forward orchestration and kernel selection.
• The CUDA backend exposes reusable continuation operations with matching Rust bindings and native ABI.
• Model-specific processors retain prompt construction, question/candidate identity and batch-layout adapters; postprocessing reconstructs outputs and normalizes across each complete question.
• Shared queue/admission policy and compatibility grouping belong to the runtime, with model-declared constraints.

Start with the existing worker/runtime integration and request-local reuse. Shared batching orchestration is still planned, so keep that implementation scoped separately from the continuation kernels.

Please show these boundaries in the plan, including buffer/state lifetimes when a caller cancels after dispatch. Resources must remain alive until submitted device work completes. This will let the optimization fit the shared architecture and support other compatible models later.

[twu3202]

Plan: request-local prefix reuse for Open-Jev

Scope of the first slice. Reuse stays inside one request, and request and response formats don't change, except that, with reuse on, metadata.prefix_cache moves from #55's independent_candidates to the reference's request_local_token_prefix fields. Independent forwards are compared with and without reuse; batching comes later, measured separately. The full forward stays as the reference.

1. Preparation: Open-Jev-9B (separate PR)

9B shares 27B's method, prompts and head, and needs four small changes, keyed by model_id so 27B keeps working: the pinned ids (Qwen/Qwen3.5-9B c202236, checkpoint 47e9668), a 9B model alias, the backbone dimensions in the head loader, and the export's head width (4096, not 5120). The export reads the temperature from the package's temperature.json and keeps main's other checks.

Against the reference (3308a15, uncached, the same PyTorch GDN path #55's validation used) with 9B's own head and temperature, every request below got the same token counts on both sides.

• M1 to M3: all 87 questions kept their decision; the largest probability difference was 0.041 (support_28).
• M4 and M5 (JevBench): 229 of 231 kept their decision. The two that changed were close calls: post against pickup at 0.403/0.390 native and 0.397/0.400 in the reference in one, and a near even split in the other (0.52/0.48 native, 0.44/0.56 in the reference). The largest difference was 0.104, on a 2,244-token four-way choice. Differences grow with prompt length: the mean is 0.012 above 2,000 tokens and 0.003 below 500.
• M4 alone: every decision held, with a largest difference of 0.068; open_jev: add native Rust/CUDA text worker #55 reported 0.020 there on 27B.

These are 9B measurements; a 27B claim needs a 27B run.

2. Workload and current cost

Frozen manifest with SHA-256 and exact token-ID counts, chat template included: M1, one choice with 2, 8, 32, 128 or 255 candidates over states of 256, 1,024 or 3,072 tokens; M2, the four examples/workflows files and community/support_28 (7 to 50 candidates); M3, score with 5 and 10 levels; M4, #55's 74 single-candidate JevBench requests (nothing to reuse); M5, the other 157 public JevBench tasks (3 to 6 candidates, ids and hashes only, as in #55). Latency: 9B, reuse off, one RTX 6000 Ada, median of three warm requests (two for M4, M5 and the requests above 200,000 tokens).

Request	Candidates	Tokens now	With reuse	Latency now
M1, 256-token state	255	79,567	8,955	14.7 s
M1, 1,024-token state	32	34,593	2,105	5.2 s
M1, 3,072-token state	255	797,392	11,770	122 s
M2, customer_service	8	4,637	1,032	0.85 s
M2, support_28 (128-token shared prefix)	50	8,195	1,650	1.87 s

M4 averages 76 ms per request (median 41 ms). M5's 157 requests take 79 s in total and would process 125,110 tokens instead of 464,274 with reuse.

On a warm card at its power limit, a single-candidate request takes 24 ms at 55 prompt tokens, 156 ms at 1,063 and 430 ms at 3,109, so even a short forward costs about 25 ms. The latencies above run 4 to 15% slower than that curve predicts. By it, ignoring branch attention over the prefix:

• reuse alone brings the 3,072 × 255 request to about 6.6 s, and adding batching to about 1.6 s;
• M5 as a whole gets about 2.2 times faster with reuse alone;
• support_28 gains almost nothing from reuse alone (1.5 s against 1.6 s on the curve; 60 short forwards replace 50 of about 165 tokens), but gets to about 0.95 s with batching within questions, since 19 of its 28 questions have one candidate. The reference's cache made it slower: 3.44 s against 1.87 s uncached.

3. Design and ownership

Same split as the reference's request_local_token_prefix at 3308a15, without its suffix batching; ownership per docs/architecture.md and src/runtime/README.md:

• Open-Jev processor: builds token IDs as today, then the layout: the longest common token-ID prefix of the request's candidates and, per question, the longest one beyond it, stopping one token short of the shortest candidate. A lone candidate shares nothing: one-candidate requests run the plain forward, and single-candidate questions get no question prefix. Prefix plus branch equals the full prompt token for token. Question and candidate identity and whole-question normalization are unchanged.
• Open-Jev executor: calls the new entry point; the scalar head stays on the CPU inside the admitted unit, with FP64 accumulation.
• Shared Qwen executor: a new, model-agnostic entry point takes the layout and returns one final hidden state per candidate, in order. It owns snapshot and branch state, runs the prefix once and then each branch, and selects kernels.
  • The snapshot (float32 GDN state, last three conv inputs per channel, prefix K/V per attention layer) is read-only once written; a question snapshot is a separate copy.
  • Branch state is private: GDN starts from the snapshot state and writes its chunk states to branch workspace, the conv reads the saved inputs as history, and attention appends branch K/V rows after the prefix, overwritten by the next branch. Positions continue from the prefix length under one causal mask.
• CUDA backend: three continuation operations, with matching Rust bindings in cuda.rs and an ABI bump coordinated with cua_s1: complete native vision and screenshot inference with GPU parity #64 and the cuda: declare the qwen3_5 backend #77 manifest: GDN prefill that reads an initial state and optionally writes the final one, the conv with history, and attention over cached K/V with a query position offset.
• Runtime: unchanged. The admitted unit is still one complete request, so no compatibility grouping yet; that and batching policy come later in the runtime, with model-declared limits.
• Lifetimes: these buffers sit next to the Qwen model's scratch and, like it, grow only for a request longer than any before. Contents live only inside one admitted closure, under the model mutex, so nothing needs invalidating. The entry point synchronizes its stream before returning, also on error, so neither a caller that cancels after dispatch (the closure still runs to the end) nor the next request can reuse a buffer that queued work still reads. Reuse runs eager; graph replay stays on the plain forward.
• Split points: the exact prefix can end inside a 64-token GDN chunk; rounding down to a chunk boundary recomputes up to 63 tokens per branch but lets GDN match the full forward bit for bit. The exact split is the default if it passes the tolerances, otherwise the aligned one.

4. Memory, from model dimensions

	Retained per prefix: GDN state + conv	Retained K/V per prefix token	Per 64-token branch: scratch + K/V
Qwen3.5-9B (24 GDN, 8 attention layers)	48 + 1.1 MiB	32 KiB	17 + 2 MiB
Qwen3.8-27B (48 GDN, 16 attention layers)	144 + 2.8 MiB	64 KiB	24 + 4 MiB

Per GDN layer, the state is value heads × 128 × 128 × float32 and the conv inputs (2 × key dim + value dim) × 3 × bfloat16; per attention layer and token, K/V is 2 × 4 KV heads × 256 × bfloat16. Branch scratch is the Scratch layout at the branch's length plus its K/V rows, inside the scratch the worker already keeps (1,024-token multiples, 266 MiB per 1,024 tokens on 9B).

For M1's 3,093-token prefix, one snapshot (all M1 needs) is about 146 MiB on 9B and 340 MiB on 27B; a question prefix adds its own GDN state, conv inputs and extra K/V, about 195 or 490 MiB in all.

5. Correctness

Kernels (backend PRs): continuation against one full forward for the conv, GDN and attention at both models' head counts; prefix lengths 1, 2, 31, 63, 64, 65, 127, 128, 129 and about 1,000; branch lengths 1, 64, 65 and 200. Expected: the conv bit-identical everywhere, having no chunks (its first test); GDN bit-identical at chunk-aligned splits; elsewhere, the existing float64-reference tolerances plus the largest difference from the unsplit kernel.

Model (executor and worker PRs, 9B): reuse on vs the full forward, same build, every candidate in M1 to M5, plus real prompts forced onto those splits. Declared now, not loosened after seeing results:

• decisions all identical (choice, noul side of 0.5, most likely score level);
• probabilities within 0.01, the threshold used for the GDN change in benchmarks/gdn/;
• head output (the scalar before temperature) within 0.035, which keeps probabilities within 0.01 at T = 1.897, since a probability moves at most max|ds| / (2T);
• final hidden state at the readout within 1e-2 relative L2, a bound the prep PR checks against the native vs reference spread.

I'll also report how many probabilities fall within 1e-3 and how many outputs are bit-identical. For scale, the reference's own cache against its uncached path (PyTorch GDN path) keeps all 84 decisions on M1 to M3 (the 128-candidate requests weren't run cached) but moves 23 answers by more than 1e-3 and 5 by more than 0.01 (largest 0.040). On M4 and M5 it changes 2 of 231 decisions and moves 20 answers by more than 0.01. A failed gate stops the run and is reported.

Also covered: permuted candidate and question order; one snapshot across all branches and a repeated request, both identical; requests A, B, A with different states, A identical both times; no common prefix, single-candidate questions, 255 candidates; a mid-request error leaving the worker usable.

6. Comparison

• Arms: one build; reuse is a startup setting, off by default until results are in; reuse off must reproduce main exactly. One worker per arm on the same GPU (9B fits twice in 48 GB), eager, requests alternated, one feasibility then two measured passes per arm.
• Clocks: under sustained load this card sits at its 300 W power cap at about 1 GHz (2,500 to 2,700 MHz idle), which favors the shorter reuse requests. Clocks, temperature, power and throttle reasons are logged per request, and clocks aren't changed.
• Metrics: mean and median latency, processed vs logical tokens, prefill calls, peak memory per worker, decisions and probabilities.
• Gates, declared now: correctness as above; at least 2 times lower mean latency on every M1 request with 8 or more candidates and a state of 1,024 tokens or more (the curve gives 3.6 to 17 times for reuse alone); at most 3% slower on M4.
• Budget: about two GPU hours on 9B; plan, manifest, raw results and summary go under benchmarks/ like benchmarks/gdn/.

7. PR sequence

1. Open-Jev-9B loading and export, with its validation.
2. Backend continuation operations, bindings and kernel tests, possibly split by operation.
3. Shared executor entry point with GPU parity tests against forward; no worker change.
4. Open-Jev layout, metadata and docs (architecture, runtime README), with the 9B checks and comparison.
5. Later, separately: candidate batching, and a 27B run.

[twu3202]

An update to the plan, with a change to the gates that needs your call and a note on #96.

GEMMs round differently with the number of rows. cuBLASLt picks an algorithm per M, so a prefix and its branch, run as separate GEMMs, don't round like one pass over the same tokens. In the 9B fused GDN input projection (synthetic inputs), row 0 differs in 6,768 of 12,352 outputs between M = 1 and M = 2. In a first shared-prefix executor, that alone moved the final hidden state by up to 1.1e-2 relative L2 on 9B against forward, above the 1e-2 gate, even where the conv, the gated delta rule and attention were bit-identical.

Change: for prefix reuse only, a GEMM handle that keeps one algorithm per weight shape for every M: cuBLASLt's choice at 64 rows, without split-K. With it, and with prefixes ending at multiples of 64 tokens (below), each shared-prefix result equals, bit for bit, a plain pass over the full prompt with the same handle (checked on sm_89). It doesn't bring reuse closer to forward: a full pass with it differs from forward by up to 1.5e-2 on 9B and 1.8e-2 on 4B on random prompts. The plain forward keeps today's algorithms, so single-candidate requests and Cua-S1 don't change. The fixed handle makes a single 9B pass 8% slower at 1,024 tokens and 18% slower at 3,072, so it isn't the default.

Split points: the plan made the exact split the default if it passed the tolerances. It doesn't: it changed the final hidden state by up to 1.25e-2 relative L2 on 4B and 1.20e-2 on 9B against a full pass with the same handle, and it made a prompt's result depend on the other candidates, since they set the common prefix. So prefixes end at a multiple of 64 tokens, the plan's fallback. That reruns up to 63 tokens per branch: a 3,072-token prefix with 255 branches of 38 tokens takes 6.1 s with reuse against 121 s for one forward per branch, and 7.2 s when the prefix ends 63 tokens past a multiple of 64 (synthetic tokens; M1's 3,093-token prefix reruns 21).

#96 adds prefix caching for JEV-VL with the same approach in the shared backend: the GDN state at 64-token boundaries, cached keys and values, and the last conv inputs. The differences from PR 2 and PR 3:

• jev_vl: add native inference and prefix caching #96 keeps prefix states across requests in a budgeted cache, for prompts that share a single-image prefix. PR 3 reuses the request prefix and each question's prefix within one request, for text prompts.
• jev_vl: add native inference and prefix caching #96 leaves the GEMM difference above, as its own comment notes, and compares with vLLM within 0.025. PR 2b and PR 3 make reuse bit-identical to a full pass with the same handle.
• jev_vl: add native inference and prefix caching #96 edits the attention and GDN kernels the existing workers use and lists their regression as not yet verified. PR 2 and PR 3 were checked to leave today's output unchanged on Cua-S1 (567 requests) and Open-Jev-9B (253).

One set of backend operations for both workers seems best, and I'm happy to adapt either way: #96's worker on PR 2's operations, or PR 3 on #96's.

Revised sequence, after #92 (9B): PR 2 (#97), continuation operations; PR 2b (#98), the fixed-algorithm GEMM handle; PR 3 (#99), the shared executor entry point; PR 4, the Open-Jev worker with the M1 to M5 checks and comparison. All three are open as drafts so you can compare them with #96, and I'll mark them ready once the backend path and the gates are settled.

Gates, which this changes. The plan gates reuse against today's forward, and in #92 I wrote that the gates stay as declared. Against forward, the hidden-state gate already fails on random prompts from the handle's rounding alone, before any reuse. Two options:

• A: gate on bit identity with a full pass under the same handle. The four comparisons with today's output on M1 to M5 are reported next to open_jev: support the Open-Jev-9B checkpoint #92's native vs reference numbers, not gated. Any difference there comes from the handle, not from reuse. open_jev: support the Open-Jev-9B checkpoint #92 found the hidden and head gates near the median of its native vs reference spread, and the handle's rounding is of that order, so I expect the head and probability gates to fail on some candidates for the same reason, and close calls can flip as 2 of 157 did on M5 in open_jev: support the Open-Jev-9B checkpoint #92.
• B: the fixed handle for every forward. Reuse would then match forward bit for bit and pass every gate, but both Qwen workers' outputs, Cua-S1's and Open-Jev's, would move from main's, and every pass would pay the 8 to 18% above.

I'd go with A. So two questions: A or B, and which backend path?