Bound Cua-S1 Graph capture work with request admission and cooldown

[Levius-Fubuki]

Purpose

Bound CUDA Graph capture work using distinct-request admission, cooldown, capture count and elapsed-work budgets, with eager fallback.

The stream-lifetime and reserved-pool-accounting fixes previously introduced here now live in #22. This branch integrates that fix and keeps policy controls as its incremental scope. Eviction, invalidation and rejection explicitly retire owned graph resources.

Stacked on #22. Incremental runtime comparison. The PR diff remains limited to runtime Python source; validation helpers, tests and generated evidence are preserved separately. This optimizes the Python multimodal worker, not the native Rust/CUDA worker.

Test Plan

System1-Omni Version / Commit: f4069e0aa4b32c533d117a7821ff7b3161c3237b.

Fresh validation started 2026-09-30 on RTX 4090 24 GiB, driver 595.71.05, Torch 2.14.0+cu130, Transformers 5.17.0 and PEFT 0.21.0; BF16 base with unmerged adapter, without FLA/causal-conv1d.

• Validation checkout and pinned setup: PYTHONPATH=src:recipe/cua_s1 python -m pytest tests/cua_s1 -q.
• Exact GPU commands and exit statuses, using upstream-verified weights and a fresh output directory.
• Runtime/validation source mapping: git diff <runtime> <validation> -- src is empty. Source hashes and numerical/performance reports are independently verified.

Test Result

• CPU suite: 187 passed in 17.76s.
• Fresh full-checkpoint tests pass five changed-image/text cases and four admission/fallback policies.
• Seven full-logit comparisons, including three changed-content survivor replays after actual eviction, are bitwise equal to eager. Retained references verify synchronous retirement, and allocator snapshots confirm reserved-pool accounting.
• The resident cache limit excludes model weights and temporary candidates; synchronous capture can overshoot its time allowance before subsequent attempts are throttled. No broad hard-process-memory or latency guarantee is claimed.
• GitHub rust CI passed on the above runtime head.

Full evidence and tradeoffs · Raw primary report · Independent verification summary.

This run validates correctness on synthetic inputs; it makes no new latency or throughput claim.

Self-review

Agent-assisted full-diff review and validation limits found no actionable correctness or architecture defect. Runtime scope, commands and claims were checked against the recorded sources/results. The contributor remains responsible for understanding the changes; this does not replace maintainer review.

[hsliuustc0106]

Review of 4862bd3fc525 (incremental stack changes).

Reviewed admission/cooldown/capture budgets, graph stream ownership, pool accounting and cleanup. No independent actionable correctness defect found in the inspected paths.

This PR addresses two substantive defects in #22: independently evictable graphs sharing a capture stream, and memory_allocated deltas omitting retained graph-pool scratch. I recommend folding those necessary lifetime/accounting fixes into #22 so the earlier PR is safe independently, rather than treating them solely as a later optimization.

Validation: 187 archived CPU tests passed against this head's runtime source. CUDA stream/pool tests use mocks. Before merging graph execution, validate actual capture, eviction, surviving-graph replay and memory accounting on the pinned GPU stack. Local Torch 2.13 / Transformers 5.14.1 differs from the recipe's 2.14 / 5.17 pins. No new latency measurement was performed.

The validation suites were retrieved from the immutable archived revisions linked in the PR descriptions; they are not retained in the current PR diffs.

[Levius-Fubuki]

@hsliuustc0106 Your request to move the stream-lifetime and reserved-pool fixes into #22 is addressed. #22 now contains those fixes at 552c02ce9ce4; #33 integrates them at f4069e0aa4b3. Please re-review #33's incremental policy delta, after #22.

Pinned-stack RTX 4090 full-checkpoint validation passes five changed-image/text cases and four admission/fallback policies. Seven full-logit comparisons are bitwise equal to eager, including three changed-content replays of a surviving graph after actual eviction. Retained references confirm explicit retirement, and pool snapshots verify reserved scratch is counted. The archived CPU suite reports 187 passes; rust CI passed on this head.

Immutable evidence and reproduction commands · admission/fallback report · eviction/pool report. The evidence/source verifier was rerun successfully on October 1. It merges cleanly with current main (566dec1), with changed files matching the validated source.

The resident cache budget excludes model weights and temporary capture candidates. Synchronous capture can exceed its time allowance before later attempts are throttled; this is not a hard process-memory or per-request latency guarantee. Proposed merge order: #17 → #18 → #22 → #33.

[hsliuustc0106]

Independent local review — graph capture admission (vs current main)

Verdict: approved. No blocking findings; one structural note for future reviewers.

Reviewed against current main (which already contains #22's multimodal merge), not the PR's old base — the PR-vs-base diff therefore reads as whole-file additions; the real delta is graph_admission.py (new) plus the graph_runtime.py integration (+124/-64 vs main), model.py (+6), and the frontend adapter. GitHub computes this PR MERGEABLE against main.

AdmissionPolicy is clean bounded-state design: LRU-capped history/cooldowns (128), sliding capture window, warmup threshold, ms-budgeted capture ledger, named eager-fallback reasons, called under the runtime lock. Eviction sets cooldown; cache hits spend no budget. This is capture policy inside the model-owned worker — exactly where it belongs under docs/architecture.md until shared runtime scheduling exists.

Disclosure: ~1.3k lines reviewed at module + structural depth (no torch on this machine, so your recorded CPU suite and RTX 4090 parity evidence are author-reported; the linked raw records, source manifest, and seven bitwise-eager logit comparisons including post-eviction survivor replays are consistent and honest about limits). rust CI green on the head.

Approval per the repo review process; reflects head f4069e0a only and asserts no completed GPU validation.