Skip to content

fix(openai): preserve Responses message phase across stateless replay (#2269) - #2295

Merged
gold-silver-copper merged 2 commits into
mainfrom
cassette-bug-hunt-20260812-r2
Aug 12, 2026
Merged

fix(openai): preserve Responses message phase across stateless replay (#2269)#2295
gold-silver-copper merged 2 commits into
mainfrom
cassette-bug-hunt-20260812-r2

Conversation

@gold-silver-copper

Copy link
Copy Markdown
Contributor

Summary

Second cassette-driven live bug hunt against the OpenAI, Anthropic, and Gemini providers (same doctrine as #2294): record new tests against the real APIs until a genuine defect surfaces, fix it, re-record what the fix invalidates.

Bug found and fixed

OpenAI Responses: the documented phase field was silently dropped from message output items and never replayed — the phase half of issue #2269.

  • Symptom, proven by traffic already in the repo: every committed multi-turn gpt-5.6 cassette shows "phase":"final_answer" on the message items in each response and no phase in any follow-up request. OpenAI documents that follow-up requests should replay it and that dropping it can degrade performance.
  • Root cause: OutputMessage had no phase field, so deserialization dropped it; nothing in the rig-message roundtrip could carry message-level wire data back to request assembly.
  • Fix (responses_api/mod.rs):
    • OutputMessage gains phase: Option<String> — the raw path now round-trips it value-equal;
    • ingest parks it in the first text block's captured wire extras under openai_responses (the same vehicle annotations already use);
    • request assembly (assistant_text_replay_message) hoists it back to the message level on Message::Assistant, filtered out of the content-level extras so it serializes exactly where the wire wants it.
  • Proof: tests/providers/openai/cassette/phase_roundtrip.rs + its cassette — a fresh two-turn recording in which turn 1's response carries phase and turn 2's recorded request body replays it on the assistant message ("id":"msg_…","phase":"final_answer","role":"assistant"), accepted by the live API. The harness matches outbound bodies, so a regression fails replay as a mock miss.
  • Scope note: streaming ingestion does not yet route item extras into params (pre-existing, documented on OPENAI_RESPONSES_EXTRAS_KEY); this fix covers the blocking path, consistent with annotations. feat(openai): preserve Responses items for stateless replay #2269's other asks (compaction input items, caller-built raw requests) overlap open PR feat(openai): support server-side Responses compaction #2234 and are deliberately not touched.

Re-recorded

gpt_5_6_reasoning/five_turn_metadata_roundtrip.yaml — the one cassette the fix invalidates. The re-recording is a 4-line diff: the four follow-up request bodies now carry phase; responses unchanged.

New coverage recorded (independent of the fix)

  • anthropic/stop_sequences — a stop_sequences hit (via additional_params) surfaces FinishReason::Stop, the raw response preserves stop_reason: "stop_sequence" + the matched stop_sequence: "five", and the emitted text stops before the sequence (claude-haiku-4-5).
  • gemini/image_input — inline base64 image ingestion (inline_data request part → grounded description); the suite previously covered only the image generation direction.

Tracker sweep

Full sweep of all open issues/PRs was done earlier today and reported in #2294; the delta since is one new PR (#2293, Mistral — out of scope). This PR addresses the in-scope follow-up ranked highest there:

Recording matrix

Scenario Provider Kind Outcome
phase_roundtrip (2-turn) openai tracker #2269 bug confirmed → fixed → recorded post-fix, pass
five_turn_metadata_roundtrip openai re-record (invalidated) 4-line diff, pass
stop_sequences anthropic coverage gap recorded, pass
image_input gemini coverage gap recorded, pass

Hygiene review done file-by-file: API keys/request_ids redacted, no secrets (two AIza-ish substrings in the gemini cassette are case-mismatched base64 image bytes, not key-shaped; the recorder's key-shape scan also passed), no unrelated captures, placeholders redact rather than fabricate. No hand-edited fixtures.

Verification

  • Baselines on this branch before any change: openai 108 / anthropic 79 / gemini 131, all green (no pre-existing failures).
  • After: cargo fmt clean; cargo clippy --all-targets --all-features clean; cargo test -p rig-core --all-features all pass; replay suites openai 109 / anthropic 80 / gemini 132 passed; wire-sharing suites also green: chatgpt 24, doubleword 21, bedrock 19.

Estimated API spend

Under $0.20: one two-turn and one five-turn gpt-5.6 recording at low reasoning effort (~2k tokens total), one claude-haiku-4-5 call (~100 tokens), one gemini-2.5-flash vision call (~1.3k input tokens).

…#2269)

gpt-5.6-era models stamp a `phase` (e.g. final_answer) on message output
items, and OpenAI documents that follow-up requests should replay it —
dropping it can degrade performance. OutputMessage silently dropped the
field at deserialization, so every replayed history omitted it (visible
in all committed multi-turn gpt-5.6 cassettes: phase in each response,
absent from each follow-up request).

The field now survives end to end: OutputMessage carries it (raw-path
serialization stays value-equal), ingest parks it in the first text
block's wire extras (the same vehicle as annotations), and request
assembly hoists it back to the message level, where the recorded live
traffic shows OpenAI accepting it.

New cassette-backed coverage recorded along the way:
- openai/phase_roundtrip: two-turn recording proving the wire emits
  phase and the follow-up request replays it (request-boundary match)
- anthropic/stop_sequences: stop_sequences hit surfaces
  FinishReason::Stop plus the matched raw stop_sequence
- gemini/image_input: inline base64 image ingestion (the suite only
  covered the generation direction)

Re-recorded gpt_5_6_reasoning/five_turn_metadata_roundtrip — the one
cassette the fix invalidates (its follow-up request bodies now carry
phase; 4-line diff).
The phase injection lived in the shared From<Output> conversion, so it
also changed the replayed request bytes of every other consumer of the
Responses wire — Copilot's recorded gpt-5.3-codex responses carry
phase, and its reasoning_roundtrip cassette failed replay as a body
mismatch. No Copilot/ChatGPT/xAI recording proves those backends accept
a replayed phase, and re-recording Copilot needs credentials this run
does not have, so capture now happens in the blocking normalize path
gated to provider == "openai" — the one backend whose acceptance the
phase_roundtrip cassette proves. The unverified wires stay
byte-identical to their recordings.
@gold-silver-copper

Copy link
Copy Markdown
Contributor Author

CI fix pushed: the phase capture originally sat in the shared From<Output> conversion, which also changed Copilot's replayed request bytes (its recorded gpt-5.3-codex responses carry phase), failing copilot::reasoning_roundtrip::nonstreaming as a cassette body mismatch. Capture is now in the blocking normalize path gated to provider == "openai" — the one backend whose acceptance the new recording proves; Copilot/ChatGPT/xAI request bytes stay identical to their recordings (no wire change without a recording to back it). Full local sweep after the fix: openai 109, copilot 69, chatgpt 65, xai 75, doubleword 62, anthropic 121, gemini 173, bedrock 73, rig-core 1325 — all green; fmt/clippy clean.

@gold-silver-copper
gold-silver-copper added this pull request to the merge queue Aug 12, 2026
Merged via the queue into main with commit bb6b6cb Aug 12, 2026
13 checks passed
@github-actions github-actions Bot mentioned this pull request Aug 12, 2026
pull Bot pushed a commit to appelgriebsch/rig that referenced this pull request Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant