FE-1657: Separate Brunch-authored Voice speech from on-screen responses

.changeset/quiet-voice-guide.md

@@ -0,0 +1,5 @@
+---
+"@hashintel/petrinaut": patch
+---
+
+Update the AI assistant guide to explain separate Brunch-authored spoken takeaways, complete on-screen reports, and explicit full-response reading.

apps/brunch-agent/src/agents/chat-agent/agent.ts

@@ -17,6 +17,7 @@ import {
 import { useBrunchAgent } from "@hashintel/brunch-agent/flue";
 
 import { ping } from "./tools/ping.ts";
+import { useVoiceResponse } from "./voice-response.ts";
 
 export const CHAT_MODEL_ID =
   process.env["BRUNCH_CHAT_MODEL"] || "claude-haiku-4-5";
@@ -38,12 +39,7 @@ export function ChatAgent() {
     "responseMode" in context &&
     context.responseMode === "voice"
   ) {
-    useInstruction(`Voice response style for this delivery only:
-Respond conversationally and concisely. Put the necessary question or conclusion first.
-Avoid unnecessary preambles and repetition; preserve consequential qualifications.
-For a short clarification, prefer one or two spoken sentences, with any consequential qualification, rather than an unsolicited report or a repeated summary. Expand only when the question requires it.
-When a detailed report is needed, keep it complete in the visible canonical response; the application offers to read long responses on request.
-These are presentation instructions only. Retain all domain, evidence, workpiece, and tool obligations.`);
+    useVoiceResponse();
   }
 
   useInstruction(

apps/brunch-agent/src/agents/chat-agent/voice-response.ts

@@ -0,0 +1,51 @@
+import {
+  defineTool,
+  useDataWriter,
+  useInstruction,
+  useTool,
+} from "@flue/runtime";
+import * as v from "valibot";
+
+import {
+  BRUNCH_VOICE_DATA_NAME,
+  BRUNCH_VOICE_TOOL_NAME,
+  BrunchVoiceDataSchema,
+  BrunchVoiceInputSchema,
+} from "@hashintel/brunch-agent/voice-response";
+
+/** App-owned delivery instructions and output, mounted only for Voice deliveries. */
+export const useVoiceResponse = (): void => {
+  const writeSpeech = useDataWriter(BRUNCH_VOICE_DATA_NAME, {
+    schema: BrunchVoiceDataSchema,
+  });
+  useTool(
+    defineTool({
+      name: BRUNCH_VOICE_TOOL_NAME,
+      description:
+        "Author the spoken answer or takeaway for this Voice reply after gathering its tool evidence. This records text, not playback or completion. Then deliver the full visible response in ordinary assistant prose. If further substantive tools are needed, replace the speech after their results.",
+      input: BrunchVoiceInputSchema,
+      output: v.object({ title: v.string(), detail: v.string() }),
+      run({ data, toolCallId }) {
+        writeSpeech({ speech: data.speech, toolCallId });
+        return {
+          output: {
+            title: "Brunch-authored speech (not playback confirmation)",
+            detail: data.speech,
+          },
+        };
+      },
+    }),
+  );
+  useInstruction(`Voice response style for this delivery only:
+Author both the spoken content and the complete visible canonical response. Realtime reads your spoken content verbatim; it does not select, summarize, or add meaning.
+For a short answer, give a brief useful answer, then ask a follow-up only when it materially advances the person's modelling goal. Do not force a follow-up every turn. Clarify first when ambiguity would materially change the answer.
+For a long analysis, author a brief substantive takeaway for speech while keeping the complete report and required recoverable workpiece in ordinary visible prose. Preserve consequential qualifications in the takeaway, not only on screen. Avoid unnecessary preambles and repetition.
+Voice delivery order (including clarification-only replies):
+1. Finish gathering the evidence and substantive tool results needed for this reply.
+2. Call brunch_set_voice_response with the exact spoken answer, takeaway, or clarification question. A clarification-only reply still needs authored speech; use the exact question text when speaking a question.
+3. If asking a direct question, call brunch_mark_question with its exact text immediately before presenting it in ordinary assistant prose. The speech tool does not replace question marking.
+4. Deliver the full visible response, including the exact marked question text, then finish. Do not skip speech authoring when the visible response is only a question.
+If another substantive tool is needed after speech authoring, replace the spoken content after that tool's result before final delivery. Do not claim a change or successful check before its evidence exists.
+The application waits for the whole reply, including browser-tool continuations, before playback. Read full response reads the complete visible prose on request. Recording speech is not evidence of playback, user agreement, or completed modelling.
+These are presentation instructions only. Retain all domain, evidence, workpiece, and tool obligations.`);
+};

apps/brunch-agent/test/architecture/boundaries.integration.ts

@@ -387,6 +387,7 @@ describe("core auxiliary subpaths stay in their assigned lanes", () => {
       "./flue",
       "./question-marker",
       "./storage",
+      "./voice-response",
       "./workpiece",
     ]);
   });
@@ -449,6 +450,8 @@ describe("the HASH smoke is runnable without a model key or a network (spec §12
       "Constructs Flue's content-free OpenTelemetry instrumentation with an injected exporter setup to prove disposal order; it registers no global instrumentation, opens no socket, and makes no provider call.",
     "apps/brunch-agent/test/voice-context.test.ts":
       "Boots the production ChatAgent with a faux provider to compare effective system prompts across typed, Voice, and browser-result deliveries — no provider key, socket, or network model call.",
+    "apps/brunch-agent/test/voice-response.test.ts":
+      "Boots the production ChatAgent with a faux provider through the HTTP router and AI SDK transport, then restarts its temporary SQLite runtime to prove speech and visible response durability — no provider key, socket, or network model call.",
     "apps/brunch-agent/test/workpiece.test.ts":
       "Types Flue's public conversation snapshot so the substrate-neutral workpiece selector and app-owned SHA-256 projection can be unit-tested against in-memory messages — no provider key, no socket, no model call, no runtime boot.",
     "libs/@hashintel/brunch-agent/packages/transport-aisdk/test/chat-transport.test.ts":

apps/brunch-agent/test/voice-context.test.ts

@@ -11,13 +11,15 @@ import { ChatAgent, CHAT_MODEL_ID } from "../src/agents/chat-agent/agent";
 
 test("ChatAgent scopes its fixed Voice instructions to the current delivery", async () => {
   const prompts: string[] = [];
+  const tools: string[][] = [];
   const provider = fauxProvider({
     provider: "anthropic",
     models: [{ id: CHAT_MODEL_ID }],
   });
   provider.setResponses(
     Array.from({ length: 5 }, () => (context) => {
       prompts.push(context.systemPrompt ?? "");
+      tools.push(context.tools?.map((tool) => tool.name) ?? []);
       return fauxAssistantMessage([fauxText("Canonical answer.")]);
     }),
   );
@@ -66,10 +68,26 @@ test("ChatAgent scopes its fixed Voice instructions to the current delivery", as
     expect(prompts[1]).toContain("Voice response style");
     expect(prompts[1]).toContain("consequential qualifications");
     expect(prompts[1]).toContain("visible canonical response");
+    // Assert presence without printing the effective prompt on failure.
+    expect(
+      prompts[1]?.includes(
+        "Voice delivery order (including clarification-only replies):",
+      ),
+    ).toBe(true);
+    expect(
+      /1\. Finish gathering[^\n]*\n2\. Call brunch_set_voice_response[^\n]*\n3\. If asking a direct question, call brunch_mark_question[^\n]*\n4\. Deliver the full visible response/u.test(
+        prompts[1] ?? "",
+      ),
+    ).toBe(true);
     expect(prompts[2]).toBe(prompts[1]);
     expect(prompts[3]).toBe(prompts[0]);
     expect(prompts[4]).toBe(prompts[0]);
     expect(prompts.join("\n")).not.toContain("UNTRUSTED_CONTEXT");
+    expect(tools[0]).not.toContain("brunch_set_voice_response");
+    expect(tools[1]).toContain("brunch_set_voice_response");
+    expect(tools[2]).toEqual(tools[1]);
+    expect(tools[3]).toEqual(tools[0]);
+    expect(tools[4]).toEqual(tools[0]);
   } finally {
     await runtime.stop();
   }

apps/brunch-agent/test/voice-response.test.ts

@@ -0,0 +1,139 @@
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import {
+  fauxAssistantMessage,
+  fauxProvider,
+  fauxText,
+  fauxToolCall,
+} from "@earendil-works/pi-ai";
+import { sqlite, start } from "@flue/runtime/node";
+import { createAgentRouter } from "@flue/runtime/routing";
+import { createFlueClient } from "@flue/sdk";
+import { expect, test } from "vitest";
+
+import {
+  createFlueChatTransport,
+  snapshotToUiMessages,
+} from "@hashintel/brunch-agent-transport-aisdk";
+import { BRUNCH_VOICE_TOOL_NAME } from "@hashintel/brunch-agent/voice-response";
+
+import { ChatAgent, CHAT_MODEL_ID } from "../src/agents/chat-agent/agent";
+
+import type { UIMessageChunk } from "ai";
+
+test("Voice speech crosses the real runtime and transport and survives restart beside the full response", async () => {
+  const speech = "  Capacity is not established. Which limit matters?  ";
+  const report =
+    "# Full analysis\n\nCapacity is not established.\n\nWhich limit matters?\n\n```runbook-ir\n# Workpiece\nTiming remains unknown.\n```";
+  const provider = fauxProvider({
+    provider: "anthropic",
+    models: [{ id: CHAT_MODEL_ID }],
+  });
+  provider.setResponses([
+    fauxAssistantMessage(
+      [fauxToolCall(BRUNCH_VOICE_TOOL_NAME, { speech }, { id: "speech-1" })],
+      { stopReason: "toolUse" },
+    ),
+    fauxAssistantMessage(
+      [
+        fauxToolCall(
+          "brunch_mark_question",
+          { question: "Which limit matters?" },
+          { id: "question-1" },
+        ),
+      ],
+      { stopReason: "toolUse" },
+    ),
+    fauxAssistantMessage([fauxText(report)]),
+  ]);
+  const directory = await mkdtemp(join(tmpdir(), "brunch-voice-response-"));
+  const boot = () =>
+    start({
+      agents: [{ agent: ChatAgent, name: ChatAgent.agentName }],
+      providers: [provider.provider],
+      db: sqlite(join(directory, "conversation.db")),
+    });
+  let runtime = await boot();
+  try {
+    const router = createAgentRouter(ChatAgent);
+    const client = createFlueClient({
+      url: "http://local.test/voice-response",
+      fetch: async (input, options) =>
+        router.fetch(
+          input instanceof Request ? input : new Request(input, options),
+        ),
+    });
+    const transport = createFlueChatTransport({
+      client,
+      clientToolNames: new Set(),
+    });
+    const stream = await transport.sendMessages({
+      chatId: "voice-response",
+      trigger: "submit-message",
+      messageId: undefined,
+      messages: [
+        {
+          id: "voice-user",
+          role: "user",
+          metadata: { source: "voice" },
+          parts: [{ type: "text", text: "Explain the limit." }],
+        },
+      ],
+      abortSignal: undefined,
+    });
+    const chunks: UIMessageChunk[] = [];
+    for await (const chunk of stream) chunks.push(chunk);
+    expect(chunks).toContainEqual({
+      type: "data-brunch-voice-response",
+      data: { speech, toolCallId: "speech-1" },
+    });
+    expect(chunks).toContainEqual({
+      type: "tool-output-available",
+      toolCallId: "speech-1",
+      output: {
+        title: "Brunch-authored speech (not playback confirmation)",
+        detail: speech,
+      },
+      providerExecuted: true,
+    });
+    expect(
+      chunks
+        .filter((chunk) => chunk.type === "text-delta")
+        .map((chunk) => chunk.delta)
+        .join(""),
+    ).toBe(report);
+    expect(chunks.at(-1)).toEqual({ type: "finish", finishReason: "stop" });
+
+    const before = snapshotToUiMessages(await client.history(), {
+      clientToolNames: new Set(),
+      hiddenToolNames: new Set(["brunch_mark_question"]),
+    });
+    const assistant = before.find((message) => message.role === "assistant");
+    expect(assistant?.parts).toContainEqual({
+      type: "data-brunch-voice-response",
+      data: { speech, toolCallId: "speech-1" },
+    });
+    expect(assistant?.parts).toContainEqual({
+      type: "text",
+      text: report,
+      state: "done",
+    });
+    expect(assistant?.parts).toContainEqual({
+      type: "data-brunch-question",
+      data: { question: "Which limit matters?", toolCallId: "question-1" },
+    });
+    await runtime.stop();
+    runtime = await boot();
+    expect(
+      snapshotToUiMessages(await client.history(), {
+        clientToolNames: new Set(),
+        hiddenToolNames: new Set(["brunch_mark_question"]),
+      }),
+    ).toEqual(before);
+  } finally {
+    await runtime.stop();
+    await rm(directory, { recursive: true, force: true });
+  }
+});

apps/petrinaut-website/README.md

@@ -124,6 +124,19 @@ bubble is replaced by the finalized message or pending-question tool output,
 which retains a waveform indicator without duplicating the answer. Provisional
 transcription and Realtime audio are not persisted as chat history.
 
+For Voice replies, Brunch authors separate spoken content and complete visible
+prose. Short replies should give a useful brief answer; long analyses should
+give a substantive spoken takeaway while keeping the complete report on screen.
+The **Brunch-authored speech (not playback confirmation)** tool result retains
+the exact authored speech alongside its response, including after reopening.
+It records authorship, not whether the audio was heard. Automatic playback waits
+for the whole correlated reply, including browser-tool continuations. Later
+substantive tools invalidate an earlier speech draft unless Brunch replaces it.
+Missing, unusable, or uncorrelated speech produces only the fixed reading notice,
+never an application-generated summary or automatic full-report reading. That
+notice is not successful substantive Voice delivery. Content usefulness,
+speech/report fidelity, and audible delay require separate human assessment.
+
 The text composer remains available. Sending typed text ends Voice mode first,
 then submits the draft exactly once through the same conversation; a failed
 handoff restores the draft. Closing the assistant pauses capture and speech
@@ -134,7 +147,7 @@ acknowledgements and response terminal event, and only then opens the
 microphone for fresh capture. Its playback menu offers **Repeat question** and
 **Read full response**. Full-response replay becomes available once the matching
 response and audio output have both finished, enqueues all exact retained
-canonical segments in order, and is disabled during capture, submission,
+visible prose segments in order—not just the spoken takeaway—and is disabled during capture, submission,
 cancellation, pause, and errors. **Repeat question** has the same safety gates
 and replays only exact question text carrying Brunch's non-interactive marker;
 if the marker is missing, malformed, or does not match finalized prose, the