Skip to content

feat: add bounded transient response sessions to core - #257

Draft
LOGO127 wants to merge 1 commit into
vllm-project:mainfrom
LOGO127:feat/core-response-sessions
Draft

feat: add bounded transient response sessions to core#257
LOGO127 wants to merge 1 commit into
vllm-project:mainfrom
LOGO127:feat/core-response-sessions

Conversation

@LOGO127

@LOGO127 LOGO127 commented Sep 6, 2026

Copy link
Copy Markdown

Summary

Related to #238; this is a draft, core-only prerequisite, not a complete WebSocket implementation or a claim that #238 is fixed.

A response may need to continue from an in-memory checkpoint without making its history durable. This adds an opt-in, transport-independent continuation boundary in the existing core executor:

  • ResponseSession, ResponseSessionGroup, and ResponseContinuation provide typed checkpoint lookup and one active execution per session. Group members can share a retained parent while keeping independent execution and failure lifetimes.
  • Preserve canonical reasoning/message/tool history across executor rounds and compaction. Validate continuation tool calls against the retained or restored history.
  • Bound retained checkpoint bytes per session and group. Account for shared snapshots once, including pinned and prepared snapshots, and reserve capacity before durable writes. Failure and cancellation release execution/reservation ownership; a failed fork preserves its source checkpoint.
  • Keep transient checkpoints separate from durable responses. Explicit Conversations retain their existing handler as the sole durable output writer. No-session HTTP and split-inference behavior remains on the existing path.

Review and integration boundary

This is based directly on current main 74c6b2d, without importing the unmerged #240 branch. It intentionally does not activate sessions in the WebSocket handler, replace #240's multiplexer, introduce another ingestion/relay path, or choose transport admission, stream-limit, and connection-expiry policies. A follow-up adapter should use the agreed transport design and preserve #240's work.

The budgets are caller-supplied retained-state limits, not a process-heap or temporary-allocation bound. Cancellation after a database COMMIT has begun cannot guarantee rollback. Those are explicit contract boundaries, not claims of complete transport compatibility.

API impact: RequestContext gains an optional continuation field. Workspace struct literals are updated; downstream literal construction will also need the field (normally None).

The proposed maintenance slice is continuation history, retention accounting, and disposal correctness. Scope/API feedback is welcome before transport integration. AI assistance was used for implementation and verification; I have personally reviewed the patch and signed the contribution.

Test Plan

Revalidated the exact signed commit 51c3613 using a fresh, isolated build directory on WSL Ubuntu x86_64 / Rust 1.98.0:

  • cargo test --locked --offline --workspace --all-features -- --test-threads=4 --quiet: 940 passed, 9 ignored across 42 test/doctest targets, excluding duplicate nested subprocess summaries. The ignored cases are eight PostgreSQL-specific tests and one existing doctest.
  • cargo clippy --locked --offline --workspace --all-targets --all-features -- -D warnings: passed.
  • Applicable pre-commit hooks on all 20 committed paths, including rustfmt, and git diff HEAD^ HEAD --check: passed.
  • The 28 public continuation scenarios cover history/compaction, own/fork isolation, explicit-Conversation output ownership, aggregate budget rejection, a real pending SQLite pool-acquisition cancellation, and transaction rollback after partial item writes.
  • Repeated the concurrent aggregate-reservation regression 100 times, cancellation regression 20 times, and partial-SQL-failure regression 20 times: all passed. These are additional executions of three existing tests, not 140 new tests.

Storage for this fresh full run was SQLite/disabled storage, with local mock/cassette upstreams. PostgreSQL is not fully green: an earlier isolated run of the preceding 27-test session suite had 25 passes (17 disabled-storage controls) and two explicit-Conversation failures. Both also fail in no-session controls because main's ConversationStore::get_or_create has a created_at SQL ambiguity (42702). #166 already contains the relevant qualification; I supplied the independent reproduction there and have not duplicated that fix here. The subsequently added partial-write rollback test passed on PostgreSQL separately, but the full 28-test PostgreSQL suite was not rerun.

No live-model/GPU, MSRV, performance, or release/platform certification is claimed.

Signed-off-by: luozijian <luozijian0924@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant