Skip to content

H-6763: Submit only completed Voice transcripts as interview answers #9500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kostandinang wants to merge 6 commits into from
Open

H-6763: Submit only completed Voice transcripts as interview answers #9500

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
947cafe
Submit only completed Voice transcripts as interview answers
kostandinang Sep 2, 2026
c4bcf19
Prevent Voice playback from becoming user input
kostandinang Sep 2, 2026
a63a712
Format Realtime transcription failure handling
kostandinang Sep 2, 2026
25f968a
Preserve Voice input started before playback
kostandinang Sep 2, 2026
5ed5a61
Recover Voice speech after rejected input
kostandinang Sep 2, 2026
935aa9f
Keep diagnostic Voice rejections passive
kostandinang Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 20 additions & 10 deletions apps/petrinaut-website/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,13 +110,22 @@ Realtime call endpoint. The provider key, model, instructions, tools, language,
and vocabulary policy stay server-side. The session uses semantic VAD with low
eagerness so natural thinking pauses are less likely to end an answer early.

Realtime is the disposable media plane: it carries continuous microphone and
remote audio, detects complete turns, and handles barge-in. Brunch remains the
control plane and sole authority for questions, captures, state, completion,
and durable history. The browser bridge accepts only the configured
`continue_interview` function, validates and serializes its arguments, rejects
duplicate or stale calls, and submits the answer through Petrinaut's shared
composer path with pending-`brunch_ask` correlation.
Realtime is the disposable media plane: it carries microphone and remote audio
and detects complete turns. Microphone capture is suspended during assistant
playback so speaker echo cannot become an interview answer, then restored to
the user's requested mute state when playback ends. Brunch remains the control
plane and sole authority for questions, captures, state, completion, and durable
history. The completed `gpt-4o-transcribe` transcript of the speaker's audio is
the only source of user answers: semantic VAD commits the audio without asking
the model to respond or interrupt playback, the session exposes no tools, and
the model never infers or paraphrases what the speaker said. The media session
drops playback-overlapping transcripts. The browser bridge normalizes
whitespace, submits each remaining completed transcript exactly once (keyed by
connection, item, and content index), ignores empty, duplicate, stale, or failed
transcripts, and submits the accepted text through Petrinaut's shared composer
path with pending-`brunch_ask` correlation and voice provenance. Silence, noise,
or a transcription failure creates no user message; the session returns to
listening and reports a recoverable "didn't catch that" notice.

The experimental **Approach D** design waits for the correlated Brunch turn,
then gives Realtime two bounded roles. First, an out-of-band, text-only request
Expand All @@ -129,9 +138,10 @@ preparation is unavailable, invalid, or times out, the bridge supplies the
canonical context and question instead. Brunch's canonical transcript and
exact question are never rewritten and remain authoritative. Preparation,
provisional transcription, and Realtime audio are ephemeral and are not
persisted. The microphone stays active while the interviewer speaks and while
Brunch is working. Speaking over assistant audio interrupts playback automatically;
WebRTC truncates provider-side unheard audio without changing Brunch history.
persisted. The microphone stays active while Brunch is working, but capture is
suspended while the interviewer speaks. Wait for playback to finish before
answering; Voice resumes listening automatically without changing Brunch
history.

The Brunch deployment must allow the website origin through its
`BRUNCH_PETRINAUT_ORIGINS` setting. Denying microphone permission leaves the
Expand Down
264 changes: 159 additions & 105 deletions apps/petrinaut-website/src/main/app/voice-interview/openai-realtime-session.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ describe("OpenAIRealtimeSession", () => {
expect(harness.peers[0]!.close).toHaveBeenCalledOnce();
});

test("keeps the microphone active through playback and reports automatic interruption", async () => {
test("does not expose audio detected during assistant playback as user speech", async () => {
const harness = createHarness();
await harness.session.connect();
harness.session.setMicrophoneEnabled(true);
Expand All @@ -253,136 +253,182 @@ describe("OpenAIRealtimeSession", () => {
item_id: "item-user",
type: "input_audio_buffer.speech_started",
});
channel.receive({
response_id: "response-canonical",
type: "output_audio_buffer.stopped",
});
channel.receive({
content_index: 0,
item_id: "item-user",
transcript: "Each bit.",
type: "conversation.item.input_audio_transcription.completed",
});
channel.receive({
content_index: 0,
item_id: "item-user",
transcript: "Each bit.",
type: "conversation.item.input_audio_transcription.completed",
});

expect(harness.localTracks[0]!.enabled).toBe(true);
expect(harness.events).toEqual(
expect.arrayContaining([
{
connectionEpoch: 1,
responseId: "response-canonical",
type: "output-started",
},
{
connectionEpoch: 1,
itemId: "item-user",
type: "input-speech-started",
},
{
connectionEpoch: 1,
responseId: "response-canonical",
type: "output-interrupted",
},
]),
);
expect(harness.events).toEqual([
{
connectionEpoch: 1,
responseId: "response-canonical",
type: "output-started",
},
{
connectionEpoch: 1,
responseId: "response-canonical",
type: "output-stopped",
},
]);
});

test("parses streamed tool arguments and the completed GA response output", async () => {
test("keeps a user turn that started before assistant playback", async () => {
const harness = createHarness();
await harness.session.connect();
harness.session.setMicrophoneEnabled(true);
harness.session.speakCanonical([
canonicalSegment("ask-1", "What happens next?"),
]);
const channel = harness.channels[0]!;
authorizeLatestSpeechResponse(channel, "response-canonical");

channel.receive({
arguments: '{"answer":"Ignored"}',
call_id: "call-ignored",
item_id: "item-ignored",
output_index: 0,
response_id: "response-tool",
type: "response.function_call_arguments.done",
audio_start_ms: 120,
item_id: "item-user",
type: "input_audio_buffer.speech_started",
});
channel.receive({
call_id: "call-1",
delta: '{"answer":"Approved"}',
item_id: "item-function",
output_index: 0,
response_id: "response-tool",
type: "response.function_call_arguments.delta",
response_id: "response-canonical",
type: "output_audio_buffer.started",
});
channel.receive({
response: {
id: "response-tool",
output: [
{
arguments: '{"answer":"Approved"}',
call_id: "call-1",
id: "item-function",
name: "continue_interview",
type: "function_call",
},
],
status: "completed",
},
type: "response.done",
content_index: 0,
item_id: "item-user",
transcript: "The supervisor approves it.",
type: "conversation.item.input_audio_transcription.completed",
});

expect(harness.events).toEqual([
{
callId: "call-1",
connectionEpoch: 1,
delta: '{"answer":"Approved"}',
itemId: "item-function",
responseId: "response-tool",
type: "tool-arguments-delta",
itemId: "item-user",
type: "input-speech-started",
},
{
arguments: '{"answer":"Approved"}',
callId: "call-1",
connectionEpoch: 1,
itemId: "item-function",
name: "continue_interview",
responseId: "response-tool",
type: "tool-arguments-done",
responseId: "response-canonical",
type: "output-started",
},
{
connectionEpoch: 1,
responseId: "response-tool",
status: "completed",
type: "response-terminal",
key: { connectionEpoch: 1, contentIndex: 0, itemId: "item-user" },
text: "The supervisor approves it.",
type: "completed",
},
]);
});

harness.session.completeFunctionCall("call-1", ["Who acts next?"]);
const [functionOutput, responseCreate] = sentEvents(channel).slice(-2);
expect(functionOutput).toEqual({
type: "conversation.item.create",
item: {
type: "function_call_output",
call_id: "call-1",
output: JSON.stringify({ response_text: ["Who acts next?"] }),
},
test("restores only the requested microphone state after assistant playback", async () => {
const harness = createHarness();
await harness.session.connect();
harness.session.setMicrophoneEnabled(true);
harness.session.speakCanonical([
canonicalSegment("ask-1", "What happens next?"),
]);
const channel = harness.channels[0]!;
authorizeLatestSpeechResponse(channel, "response-canonical");

channel.receive({
response_id: "response-canonical",
type: "output_audio_buffer.started",
});
expect(responseCreate).toMatchObject({
type: "response.create",
expect(harness.localTracks[0]!.enabled).toBe(false);

channel.receive({
response_id: "response-canonical",
type: "output_audio_buffer.stopped",
});
expect(harness.localTracks[0]!.enabled).toBe(true);
channel.receive({
response: {
instructions:
"Speak only the response_text strings supplied by Petrinaut, in array order and verbatim. Deliver them as a warm, calm, curious, confident, concise, and professionally neutral expert interviewer, at a measured conversational pace with natural emphasis. Never sound robotic, fawning, rushed, overenthusiastic, or patronizing. Do not add, remove, paraphrase, acknowledge, or explain anything.",
output_modalities: ["audio"],
parallel_tool_calls: false,
tool_choice: "none",
tools: [],
id: "response-canonical",
output: [],
status: "completed",
},
type: "response.done",
});

const responseCreateCount = sentEvents(channel).filter(
({ type }) => type === "response.create",
).length;
harness.session.completeFunctionCall(
"call-2",
["Response cancelled before speech."],
{ speakResponse: false },
);
expect(sentEvents(channel).at(-1)).toEqual({
type: "conversation.item.create",
item: {
type: "function_call_output",
call_id: "call-2",
output: JSON.stringify({
response_text: ["Response cancelled before speech."],
}),
harness.session.speakCanonical([
canonicalSegment("ask-2", "And after that?"),
]);
authorizeLatestSpeechResponse(channel, "response-canonical-2");
channel.receive({
response_id: "response-canonical-2",
type: "output_audio_buffer.started",
});
harness.session.setMicrophoneEnabled(false);
channel.receive({
response_id: "response-canonical-2",
type: "output_audio_buffer.stopped",
});

expect(harness.localTracks[0]!.enabled).toBe(false);
});

test("never surfaces model-generated function-call arguments as user speech", async () => {
const harness = createHarness();
await harness.session.connect();
const channel = harness.channels[0]!;

channel.receive({
arguments: '{"answer":"hi"}',
call_id: "call-legacy",
item_id: "item-legacy",
output_index: 0,
response_id: "response-legacy",
type: "response.function_call_arguments.done",
});
channel.receive({
call_id: "call-legacy",
delta: '{"answer":"hi"}',
item_id: "item-legacy",
output_index: 0,
response_id: "response-legacy",
type: "response.function_call_arguments.delta",
});
expect(harness.events).toEqual([]);

channel.receive({
response: {
id: "response-legacy",
output: [
{
arguments: '{"answer":"hi"}',
call_id: "call-legacy",
id: "item-legacy",
name: "continue_interview",
status: "completed",
type: "function_call",
},
],
status: "completed",
},
type: "response.done",
});

expect(harness.events).toEqual([
expect.objectContaining({ code: "invalid-response", type: "error" }),
]);
expect(JSON.stringify(harness.events)).not.toContain('"hi"');
expect(
sentEvents(channel).filter(({ type }) => type === "response.create"),
).toHaveLength(responseCreateCount);
sentEvents(channel).some(
({ item }) =>
(item as { type?: unknown } | undefined)?.type ===
"function_call_output",
),
).toBe(false);
expect(harness.localTracks[0]!.stop).toHaveBeenCalledOnce();
});

test("renders prepared strings through the verbatim out-of-band audio response", async () => {
Expand Down Expand Up @@ -1129,7 +1175,7 @@ describe("OpenAIRealtimeSession", () => {
expect(harness.peers[0]!.close).toHaveBeenCalledOnce();
});

test("treats transcripts as display-only and never closes capture", async () => {
test("emits keyed transcripts verbatim and never closes capture", async () => {
const harness = createHarness();
await harness.session.connect();
harness.session.setMicrophoneEnabled(true);
Expand All @@ -1147,6 +1193,12 @@ describe("OpenAIRealtimeSession", () => {
transcript: "The supervisor approves it.",
type: "conversation.item.input_audio_transcription.completed",
});
channel.receive({
content_index: 0,
item_id: "item-silence",
transcript: "",
type: "conversation.item.input_audio_transcription.completed",
});

expect(harness.events).toEqual([
{
Expand All @@ -1159,6 +1211,11 @@ describe("OpenAIRealtimeSession", () => {
text: "The supervisor approves it.",
type: "completed",
},
{
key: { connectionEpoch: 1, contentIndex: 0, itemId: "item-silence" },
text: "",
type: "completed",
},
]);
expect(harness.localTracks[0]!.enabled).toBe(true);
});
Expand Down Expand Up @@ -1218,13 +1275,10 @@ describe("OpenAIRealtimeSession", () => {

await expect(harness.session.connect()).resolves.toBe(2);
firstChannel.receive({
arguments: '{"answer":"Stale"}',
call_id: "call-stale",
content_index: 0,
item_id: "item-stale",
name: "continue_interview",
output_index: 0,
response_id: "response-stale",
type: "response.function_call_arguments.done",
transcript: "Stale answer",
type: "conversation.item.input_audio_transcription.completed",
});

expect(harness.events).toEqual([]);
Expand Down
Loading
Loading