From 79505701a23378520f11e62edee1a8568bdc7161 Mon Sep 17 00:00:00 2001 From: tt-a1i Date: Wed, 9 Sep 2026 01:47:29 +0800 Subject: [PATCH 1/2] fix(subagents): keep print-host parents from settling without wait --print/--json cannot re-invoke after agent_settled, so spawn text that tells the model to end the turn drops required child results. Require subagent_wait on non-TUI hosts and send a bounded follow-up barrier. --- .../CURSOR_RELIABILITY_BASELINE_2026-09-08.md | 122 ++++++++++++++++++ .../CURSOR_RELIABILITY_PROGRESS_2026-09-08.md | 83 ++++++++++++ .../CURSOR_RELIABILITY_SUMMARY_2026-09-08.md | 44 +++++++ docs/research/README.md | 6 + extensions/subagents/index.ts | 42 +++++- extensions/subagents/src/print-host.ts | 36 ++++++ extensions/subagents/src/prompt.ts | 9 +- tests/extensions/subagents/print-host.test.ts | 37 ++++++ tests/extensions/subagents/prompt.test.ts | 29 ++++- 9 files changed, 397 insertions(+), 11 deletions(-) create mode 100644 docs/research/CURSOR_RELIABILITY_BASELINE_2026-09-08.md create mode 100644 docs/research/CURSOR_RELIABILITY_PROGRESS_2026-09-08.md create mode 100644 docs/research/CURSOR_RELIABILITY_SUMMARY_2026-09-08.md create mode 100644 extensions/subagents/src/print-host.ts create mode 100644 tests/extensions/subagents/print-host.test.ts diff --git a/docs/research/CURSOR_RELIABILITY_BASELINE_2026-09-08.md b/docs/research/CURSOR_RELIABILITY_BASELINE_2026-09-08.md new file mode 100644 index 00000000..cfec62ee --- /dev/null +++ b/docs/research/CURSOR_RELIABILITY_BASELINE_2026-09-08.md @@ -0,0 +1,122 @@ +--- +status: draft +created: 2026-09-08 +last-verified: 2026-09-08 +applies-to: isolated v0.8.1 worktree a3edee28 plus local OPENPI_CURSOR_WIRE_LOG instrumentation; not the user's daily runtime +related-issues: "#234" +related-prs: "#484, #485" +supersedes: none +--- + +# OpenPI Cursor reliability experiment baseline + +Exploratory research setup and first live samples. Not a formal Benchmark. + +## Question + +Can parent-agent and subagent tasks on Cursor Grok configs be systematically accepted against machine-checkable acceptors on released OpenPI v0.8.1, and which failure classes are OpenPI-fixable? + +## Observed facts + +### User checkout (untouched) + +Commands run in `/Users/tushaokun/work/my-pi-setup` without writing to it: + +- Branch: `codex/knowledge-evidence-contract` +- HEAD: `a9b40f0044ee59c360a6077c1c0bdbdbd30da10b` +- Dirty: yes (docs/research, architecture visuals, `.hive/`, repro scripts). Left intact. +- Local `main`: `2a69d3f32994da4123f1312b7fa84ef3d6119be1` (stale vs origin) +- `origin/main`: `0d17f4577fe31315fe6c95370d251bdb4e2413cf` (contains v0.8.1) + +### Release identity + +- Annotated tag `v0.8.1` → `a3edee28fec813582db6b3b1341754cef4597f81` (`chore(release): bump version to 0.8.1 (#485)`, 2026-09-08) +- Package version at that commit: `0.8.1` +- Installed Pi CLI: `0.85.1` (`/Users/tushaokun/.bun/bin/pi`) + +### Unique daily OpenPI source (do not use for these experiments) + +`pi list` (user config, no `PI_CODING_AGENT_DIR`): + +``` +User packages: + npm:pi-web-access + /Users/tushaokun/.pi/agent/npm/node_modules/pi-web-access + ../../work/openpi-main-runtime + /Users/tushaokun/work/openpi-main-runtime +``` + +One OpenPI source: `/Users/tushaokun/work/openpi-main-runtime` at `ed9dbc1018f890fd54375f5371990ddfee8af5df`, package `0.7.0`, branch `main` behind origin by 1. Not v0.8.1. User `settings.json` packages were unchanged after isolated install. + +### Isolated experiment assets + +| Asset | Path | +| --- | --- | +| Worktree | `/tmp/openpi-reliability-20260908/src` on `research/openpi-cursor-reliability-20260908` @ `a3edee28` | +| Isolated Pi agent dir | `/tmp/openpi-reliability-20260908/pi-agent` via `PI_CODING_AGENT_DIR` | +| Synthetic git repo | `/tmp/openpi-reliability-20260908/synthetic` (HEAD `509b2472d17f50411916ee77c44ec09301b129e4`) | +| Harness | `/tmp/openpi-reliability-20260908/harness` | +| Results | `/tmp/openpi-reliability-20260908/results` | +| Continuation | `/tmp/openpi-reliability-20260908/CONTINUATION.md` | + +Isolation probe: `PI_CODING_AGENT_DIR` empty dir listed `No packages installed`; user packages and 939 user Sessions unchanged. + +Isolated `pi install` of the worktree then listed only: + +``` +User packages: + ../src + /tmp/openpi-reliability-20260908/src +``` + +Credentials: isolated `auth.json` is a copy of the user file (providers present: cursor, kimi-code, openai-codex, xai). User `auth.json` / `models-store.json` / `settings.json` mtimes were not changed by isolated `pi update --models` or catalog seed. + +### Cursor Grok config availability + +Targets: `cursor-grok-4.6-high`, `cursor-grok-4.6-high-fast`, `cursor-grok-4.6-medium-fast`. No substitutions. + +| Probe | Result | +| --- | --- | +| User `pi --list-models cursor-grok` | All three IDs present (account catalog, 217 Cursor models, `checkedAt` 1788861139372) | +| Isolated `pi update --models` | Refreshed built-in `openai-codex` and `xai` only. No `cursor` key written. | +| Isolated `pi --list-models cursor` before seed | Only static `cursor/default` from OpenPI | +| Isolated `fetchCursorUsableModels` | Failed: `discovery_transport_or_protocol_failure` in 1141 ms (`results/live-discovery.json`) | +| Isolated `pi --list-models cursor-grok` after seeding the three IDs + `default` from the user catalog | All three IDs listed | +| `pi auth check --provider cursor` | `provider_not_found` in both user and isolated CLIs (Cursor is an OpenPI-registered provider, not a Pi built-in auth provider) | + +Availability for selection in the isolated CLI is therefore: **listed after catalog seed**, **not confirmed by live GetUsableModels**. First live AgentService run (below) did stream on `cursor-grok-4.6-high-fast`, which is stronger than list-models alone for that one ID. + +### First live parent sample (cell A) + +- Run: `results/cells/A-cursor-grok-4.6-high-fast-01-1788881803136` +- Model: `cursor-grok-4.6-high-fast`, thinking `high`, tools `read,bash,rg,fd` +- Duration: 180059 ms, harness SIGTERM, exit 143 +- Ledger classification: `timeout` +- Acceptor `A-read-search-bash` v `2026-09-08.1`: `ACCEPT_A.txt` missing; nonce file and forbidden file unchanged +- Native wire log: none (no `OPENPI_CURSOR_WIRE_LOG` events) +- Discriminative evidence: 24 turns, 46 successful Pi tool executions (`read` 23, `rg` 23, 0 errors). The model repeatedly stated it had used Cursor-native / wrong wrappers and must switch to "openpi MCP tools", then called `read`+`rg` again. It never called `bash` and never wrote `ACCEPT_A.txt`. +- Refined class: **model strategy failure** that the harness cut off as **timeout**. Not an agent connection timeout, not a remote disconnect, not a native-rejection hang. +- Likely related OpenPI text: `CURSOR_PI_TOOLS_SYSTEM_PROMPT` tells the model to use "openpi MCP tools" and not Cursor-native tools. Advertised names remain `read` / `rg`. Whether that wording induces the loop is **under investigation**. + +### Limits + +- Matrix not complete. One parent cell, one model, one attempt. +- Live usable-model discovery failed; catalog seed is not a substitute for GetUsableModels. +- User daily runtime remains OpenPI 0.7.0 at `ed9dbc10`. These runs do not accept that runtime. +- Isolated worktree has uncommitted env-gated `OPENPI_CURSOR_WIRE_LOG` instrumentation in `extensions/ai-providers/cursor/provider.ts`. Behavior is unchanged when the env var is unset. +- Formal Benchmark identities are not claimed. + +## Inferences + +The v0.8.1 native-rejection path was not exercised in this A sample. The blocking live failure seen so far is a tool-identity loop after successful Pi MCP calls. + +## Unknowns + +- Whether high / medium-fast show the same loop +- Whether a prompt naming the actual tool IDs stops the loop +- Why isolated GetUsableModels failed while AgentService/Run streamed +- Cancel / disconnect / timeout / cleanup deterministic suite not yet re-run in this worktree this session + +## Next + +See `/tmp/openpi-reliability-20260908/CONTINUATION.md`. diff --git a/docs/research/CURSOR_RELIABILITY_PROGRESS_2026-09-08.md b/docs/research/CURSOR_RELIABILITY_PROGRESS_2026-09-08.md new file mode 100644 index 00000000..ed412d75 --- /dev/null +++ b/docs/research/CURSOR_RELIABILITY_PROGRESS_2026-09-08.md @@ -0,0 +1,83 @@ +--- +status: draft +created: 2026-09-08 +last-verified: 2026-09-08 +applies-to: isolated v0.8.1 worktree a3edee28 plus print-host wiring and unused-unless-set OPENPI_CURSOR_WIRE_LOG; exploratory, not a Benchmark +related-issues: "#234, #512, #513, #514" +related-prs: "#484" +supersedes: none +--- + +# OpenPI Cursor reliability — session progress + +Companion to [`CURSOR_RELIABILITY_BASELINE_2026-09-08.md`](CURSOR_RELIABILITY_BASELINE_2026-09-08.md). Ledger: `/tmp/openpi-reliability-20260908/results/ledger.jsonl`. Not a formal Benchmark. + +## Matrix (first 5 attempts per cell; all attempts kept) + +| Config | A | B | C | D | E | F | +| --- | --- | --- | --- | --- | --- | --- | +| cursor-grok-4.6-high-fast | 5 (3 correct, 2 identity-loop timeout) | 5/5 | 5/5 native≥1 | 5 (4 correct, 1 error-retry timeout) | first 5: 2 correct, 3 print-settle; later 15 after-fix all correct | 5/5 | +| cursor-grok-4.6-high | 5/5 | 5/5 | 5 (3 correct+native, 1 wrong+native, 1 no-native) | 5 (4 correct, 1 timeout) | 5/5 | 5/5 (+5 extra correct) | +| cursor-grok-4.6-medium-fast | 5 (4 correct, 1 hang-after-success timeout; acceptor still green) | 5 (4 correct, 1 hang-after-success timeout; acceptor still green) | 5 (4 correct+native, 1 timeout no-native) | 5 (2 correct, 3 timeout) | 5/5 | 5/5 | + +Class C counts only when `nativeRejections ≥ 1`. high-fast C is 5/5 valid recoveries (PR #484 live revalidation, one config, n=5). + +## Long tasks (≥10 min wall + independent external acceptor) + +| Run | Duration | Acceptor | Class | +| --- | --- | --- | --- | +| LP-high-fast-01 | 789s | pass (4 shard path hashes) | correct completion | +| LH-medium-fast-01 | 633s | fail (count/hash mismatch) | wrong result | +| LH-high-01 | 819s | fail | wrong result | +| LD-medium-fast-01 | 791s | fail (15–16 line / hash drift) | wrong result | +| LD-high-fast-01 | 1165s | fail | wrong result | +| LD-high-01 | 1710s | fail (15/16 lines, hash drift) | wrong result | + +Shorter audits (L 210s, other LPs 217–487s) have acceptors but do **not** count toward the six 10-minute tasks. + +Content-hash long tasks often mismatch because models include an extra `.ts` file or hash a different canonicalization. That is model-strategy / wrong-result, not infrastructure. One path-list pipeline passed. + +## Criterion 3 — cancel / disconnect / timeout / cleanup + +- Unit: `pi-backend-lifecycle` 20/20 (cancel, quiet thinking >45s, bounded cleanup). `cursor.test` 30/30 (abort, EOF, server cancel ≠ caller cancel, native reject then Pi roundtrip). +- Live cancel `cancel-live-1788887430222`: `sawTool=true`, SIGTERM after first `tool_execution_start`, leftover `[]`, user Sessions 939, class `cancel`. +- Live disconnect `disconnect-live-1788887477808`: isolated CONNECT proxy dropped after first tool; next stream `stopReason: error` / `Client network socket disconnected before secure TLS connection was established`; then `agent_settled`; leftover `[]`; Sessions 939. Class `infrastructure failure`. +- After harness SIGTERM loops: no leftover `run-cell` processes. Isolation leak from an earlier un-prefixed lifecycle test was removed; Sessions restored to 939. + +## Failure classes (all attempts kept) + +1. **Tool-identity loop** (A high-fast 01, 04) — [#513](https://github.com/openpi-dev/openpi/issues/513). Many successful Pi `read`/`rg`; model claims native / `mcp_openpi_*` names; never writes ACCEPT_A. Under investigation. Not connection timeout. +2. **Hang after acceptor success** (A/B medium-fast) — work written, session does not exit; harness timeout. Hung-task reliability. +3. **Error-recovery loop** (D timeouts) — repeated failed missing-path reads; never ACCEPT_D. Model strategy. +4. **Print-mode parent settle** (E high-fast 01–03) — [#512](https://github.com/openpi-dev/openpi/issues/512). Spawn then `stop` without wait; `--print` emits `agent_settled`. **Fixed in isolated worktree** (see below). +5. **Native reject without task recovery** (C high 02) — rejection recorded, zero Pi tools, wrong result. +6. **Insufficient evidence** (C high 03; first live cancel) — no tools / no native frames. +7. **Long-task hash canonicalization** — systematic extra-file or digest drift on content-hash shards. +8. **Explorer inherits write** — [#514](https://github.com/openpi-dev/openpi/issues/514). Advertised `write`/`bash` on a “read-only” role. No `CHILD_ESCAPED.txt` in E samples. + +## Print-host fix (#512) + +Before (stock v0.8.1 spawn text), same E task+acceptor, `cursor-grok-4.6-high-fast`, n=10: **7/10** correct, **3/10** print-settle wrong result. + +After (print-host spawn text + bounded `followUp` barrier), same task+acceptor, n=10: **10/10** correct. E-11 spawn result: `This host cannot re-invoke you after you end the turn. You MUST call subagent_wait...` then the model called Pi `subagent_wait` (after a brief `mcp_openpi_subagent_wait` thought — #513 residue). + +`bun run check` and `bun run test` in the isolated worktree: pass (1470 node tests + 113 vitest; 1 skipped). + +No tools disabled, permissions unchanged, no timeout lengthening, no swallowed errors. + +## Issues + +- [#512](https://github.com/openpi-dev/openpi/issues/512) print-mode settle — evidenced; fix in isolated worktree, not merged. +- [#513](https://github.com/openpi-dev/openpi/issues/513) Cursor prompt vs advertised names — under investigation; no 10/10 A-loop before-after yet (only 2 failing / 5 on high-fast A). +- [#514](https://github.com/openpi-dev/openpi/issues/514) explorer inherit write — under investigation; no proven write escape. + +## Sample limits + +- Baseline cells n=5. E high-fast n=20 for the #512 before/after only. +- Cursor usage fields are typically zeros / unknown. +- GetUsableModels probe without `ALL_PROXY` failed; live `pi --print` with the user proxy inherited streamed all three Grok IDs. +- Concurrent parents: 3 long heavies in parallel produced hash errors, not connection storms. + +## Isolation + +Always `PI_CODING_AGENT_DIR=/tmp/openpi-reliability-20260908/pi-agent`. Unique isolated OpenPI: `/tmp/openpi-reliability-20260908/src`. User packages (`pi-web-access` + `openpi-main-runtime`) and 939 Sessions unchanged. diff --git a/docs/research/CURSOR_RELIABILITY_SUMMARY_2026-09-08.md b/docs/research/CURSOR_RELIABILITY_SUMMARY_2026-09-08.md new file mode 100644 index 00000000..d6f8fad4 --- /dev/null +++ b/docs/research/CURSOR_RELIABILITY_SUMMARY_2026-09-08.md @@ -0,0 +1,44 @@ +--- +status: draft +created: 2026-09-08 +last-verified: 2026-09-08 +applies-to: isolated OpenPI v0.8.1 experiments; exploratory, not a Benchmark +related-issues: "#512, #513, #514" +related-prs: "#484" +supersedes: none +--- + +# OpenPI Cursor reliability — summary + +Exploratory. Not a formal Benchmark. Full matrix and classifications: [`CURSOR_RELIABILITY_PROGRESS_2026-09-08.md`](CURSOR_RELIABILITY_PROGRESS_2026-09-08.md). + +## What was verified + +- **Runtime:** Pi 0.85.1; experiment source v0.8.1 `a3edee28`; unique isolated package `/tmp/openpi-reliability-20260908/src`. Daily OpenPI remains `openpi-main-runtime` @ `ed9dbc10` (0.7.0) — unused. +- **Configs:** `cursor-grok-4.6-high`, `cursor-grok-4.6-high-fast`, `cursor-grok-4.6-medium-fast` all listed and all streamed live. No silent substitution. +- **18 baseline cells × 5** on the three configs (A–F). high-fast C is 5/5 native-reject → Pi recover (PR #484). +- **Six 10+ minute tasks** with external acceptors (LP-high-fast-01 path-hash pass at 789s; five content-hash runs 633–1710s with evaluated wrong-result acceptors). +- **Cancel / disconnect / timeout / cleanup:** unit 20+30; live cancel with tool-seen; live proxy-drop TLS disconnect; leftover processes empty; user Sessions 939. +- **#512 fix:** E high-fast 7/10 → 10/10 on the same acceptor. Isolated `bun run check` + `bun run test` green. + +## What remains unreliable + +- high-fast class A identity loop (~2/5). [#513](https://github.com/openpi-dev/openpi/issues/513). No 10/10 before-after yet. +- medium-fast hang-after-success (acceptor green, process does not exit). +- D error-retry loops (especially medium-fast 3/5 timeout). +- C on high: 3/5 valid native recoveries. +- Long content-hash acceptors (model canonicalization). +- Explorer/reviewer inherit write/bash. [#514](https://github.com/openpi-dev/openpi/issues/514). + +## Before / after (#512 only) + +| Source | Task | n | Correct | +| --- | --- | --- | --- | +| v0.8.1 stock spawn text | E / high-fast | 10 | 7 | +| v0.8.1 + print-host wiring | E / high-fast | 10 | 10 | + +Other cells were not re-run after the fix except in-flight longs. + +## Sample limits + +n=5 per baseline cell. Usage unknown. Live discovery without proxy failed. Do not treat this file as a Benchmark. diff --git a/docs/research/README.md b/docs/research/README.md index 25df6042..7484315e 100644 --- a/docs/research/README.md +++ b/docs/research/README.md @@ -10,6 +10,12 @@ Research records preserve sourced investigation and distinguish observations, in - [`CURSOR_NATIVE_RECOVERY_2026-09-08.md`](CURSOR_NATIVE_RECOVERY_2026-09-08.md) — bounded in-band native execution rejection, Pi-owned tools, cancellation identity, and actual child acceptance ([#234](https://github.com/openpi-dev/openpi/issues/234)). +- [`CURSOR_RELIABILITY_BASELINE_2026-09-08.md`](CURSOR_RELIABILITY_BASELINE_2026-09-08.md) — isolated v0.8.1 experiment baseline, Cursor Grok availability probes, and first parent-cell evidence (exploratory; not a Benchmark). + +- [`CURSOR_RELIABILITY_PROGRESS_2026-09-08.md`](CURSOR_RELIABILITY_PROGRESS_2026-09-08.md) — live matrix progress, failure classification, #512 before/after, and remaining cells (exploratory; not a Benchmark). + +- [`CURSOR_RELIABILITY_SUMMARY_2026-09-08.md`](CURSOR_RELIABILITY_SUMMARY_2026-09-08.md) — configs/tasks verified, remaining unreliability, before/after deltas, sample limits (exploratory; not a Benchmark). + - [`WEB_STARTUP_2026-09-07.md`](WEB_STARTUP_2026-09-07.md) — terminal startup feedback, browser-launch waiting, and exploratory timing limits ([#450](https://github.com/openpi-dev/openpi/issues/450)). - [`WEB_STREAMING_MARKDOWN_2026-09-07.md`](WEB_STREAMING_MARKDOWN_2026-09-07.md) — unchanged historical Markdown parsing during streaming, bounded React component reuse and deterministic regression evidence ([#434](https://github.com/openpi-dev/openpi/issues/434)). diff --git a/extensions/subagents/index.ts b/extensions/subagents/index.ts index f753c77b..f058198d 100644 --- a/extensions/subagents/index.ts +++ b/extensions/subagents/index.ts @@ -60,8 +60,8 @@ import { inheritedChildToolAllowlist, resolveStandaloneChildProjectTrust, } from "../shared/child-session.ts"; -import { formatContextUtilization } from "../shared/context-utilization.ts"; import { completionOwnerFor } from "../shared/completion-inbox.ts"; +import { formatContextUtilization } from "../shared/context-utilization.ts"; import { registerEditorLayer, removeEditorLayer, @@ -117,6 +117,11 @@ import { subagentIdWatermark, } from "./src/id-sequence.ts"; import { SubagentManager, type SubagentManagerShape } from "./src/manager.ts"; +import { + buildPrintHostPendingFollowUp, + canDeliverLaterFromHost, + printHostNeedsFollowUp, +} from "./src/print-host.ts"; import { buildSubagentResultDisplayMessage, buildSubagentResultMessage, @@ -145,8 +150,8 @@ import { createSubagentResultDelivery } from "./src/result-delivery.ts"; import { createSubagentRuntime, runTool, - SubagentToolInterruptedError, type SubagentRuntime, + SubagentToolInterruptedError, } from "./src/runtime.ts"; import { openSubagentPicker, openSubagentTakeover } from "./src/ui/takeover.ts"; import { @@ -547,7 +552,33 @@ export default function ( // delivery coordinator batches results that settled while it was busy. deliver: dispatchResults, }); - pi.on("agent_settled", () => resultDelivery.parentSettled()); + const runningDirectIds = new Set(); + let printHostBarrierCount = 0; + const hostCanDeliverLater = () => + sessionContext + ? canDeliverLaterFromHost({ + hasUI: sessionContext.hasUI, + mode: sessionContext.mode, + }) + : true; + pi.on("agent_settled", () => { + resultDelivery.parentSettled(); + if ( + !printHostNeedsFollowUp(hostCanDeliverLater(), runningDirectIds.size) || + printHostBarrierCount >= 2 + ) { + return; + } + printHostBarrierCount += 1; + pi.sendMessage( + { + customType: "subagent-print-barrier", + content: buildPrintHostPendingFollowUp([...runningDirectIds]), + display: false, + }, + { deliverAs: "followUp", triggerTurn: true }, + ); + }); const registerStableToolFamily = () => patchOwnedTools(pi, "subagents", { enable: OPENPI_TOOL_SURFACE.subagents.entry, @@ -708,6 +739,7 @@ export default function ( }; const onSettled = (snap: SubagentSnapshot, consumed: boolean) => { + if (snap.origin !== "btw") runningDirectIds.delete(snap.id); // A shutdown can settle children while disposing their scopes. Never // append into a session whose extension runtime is already closing. if (!sessionContext) return; @@ -745,6 +777,8 @@ export default function ( registerStableToolFamily(); sessionContext = ctx; settledAcknowledgedAt = 0; + printHostBarrierCount = 0; + runningDirectIds.clear(); if (ctx.hasUI) ui = ctx.ui; installSubagentNavigation(ctx); updateSubagentWidget(); @@ -1033,6 +1067,7 @@ export default function ( throw error; } persistId(snap.id); + if (snap.origin !== "btw") runningDirectIds.add(snap.id); return { content: [ @@ -1044,6 +1079,7 @@ export default function ( harness, modelLabel: snap.meta.modelLabel ?? "?", cwd: childCwd, + canDeliverLater: hostCanDeliverLater(), ...(worktree ? { worktreeBranch: worktree.branch } : {}), ...(agentType ? { agentTypeName: agentType.name } : {}), ...(childTools ? { tools: childTools } : {}), diff --git a/extensions/subagents/src/print-host.ts b/extensions/subagents/src/print-host.ts new file mode 100644 index 00000000..84ae139e --- /dev/null +++ b/extensions/subagents/src/print-host.ts @@ -0,0 +1,36 @@ +/** + * Print / RPC hosts cannot re-invoke the parent after agent_settled. + * Workflows already wait on non-delivery hosts; subagent spawn text still + * tells the model it may end the turn. + */ + +export function canDeliverLaterFromHost(host: { + hasUI?: boolean; + mode?: string; +}): boolean { + return host.hasUI === true && host.mode === "tui"; +} + +export function printHostNeedsFollowUp( + canDeliverLater: boolean, + runningDirectCount: number, +): boolean { + return !canDeliverLater && runningDirectCount > 0; +} + +export function buildPrintHostPendingFollowUp(ids: readonly string[]): string { + const listed = ids.join(", "); + return ( + `This is a single-invocation non-interactive host. Subagents still running: ${listed}. ` + + `You MUST call subagent_wait(ids: ${JSON.stringify(ids)}) now. ` + + `Ending the turn loses their results; there is no automatic re-invoke.` + ); +} + +export function buildNonInteractiveSpawnContinuation(id: string): string { + return ( + `This host cannot re-invoke you after you end the turn. ` + + `You MUST call subagent_wait(ids: ["${id}"]) before ending. ` + + `Do not end the turn while this child is running.` + ); +} diff --git a/extensions/subagents/src/prompt.ts b/extensions/subagents/src/prompt.ts index ab74567b..70a73d3f 100644 --- a/extensions/subagents/src/prompt.ts +++ b/extensions/subagents/src/prompt.ts @@ -7,6 +7,7 @@ import { SUBAGENT_ROLE_NAMES } from "../../shared/subagent-roles.ts"; import { type AgentType, READ_ONLY_AGENT_TOOLS } from "./agent-types.ts"; import { BACKEND_NAMES, REASONING_EFFORTS } from "./domain.ts"; import { MAX_RUNNING } from "./manager.ts"; +import { buildNonInteractiveSpawnContinuation } from "./print-host.ts"; export const SUBAGENT_SCHEMA_BUDGETS = Object.freeze({ rolePurposeBytes: 240, @@ -218,6 +219,8 @@ export function buildSubagentSpawnResult(options: { tools?: readonly string[]; worktreeBranch?: string; structured?: boolean; + /** False for print/RPC hosts that cannot re-invoke after agent_settled. */ + canDeliverLater?: boolean; }) { const typeNote = options.agentTypeName ? ` Agent type "${options.agentTypeName}" applied.` @@ -240,8 +243,12 @@ export function buildSubagentSpawnResult(options: { const structuredNote = options.structured ? " This run must finish with the requested validated structured result." : ""; + const header = `Spawned subagent ${options.id} "${options.title}" (${options.harness}: ${options.modelLabel}, ${options.cwd}).${typeNote}${toolNote}${worktreeNote}${structuredNote}\n`; + if (options.canDeliverLater === false) { + return `${header}${buildNonInteractiveSpawnContinuation(options.id)}`; + } return ( - `Spawned subagent ${options.id} "${options.title}" (${options.harness}: ${options.modelLabel}, ${options.cwd}).${typeNote}${toolNote}${worktreeNote}${structuredNote}\n` + + header + `It runs in the background — keep working on independent work. If none remains in an interactive session, briefly tell the user it is still running and end your turn; its result is delivered automatically and you are automatically re-invoked when it finishes. Do not poll or call subagent_wait merely because a later step depends on it. ` + `Use subagent_wait(ids: ["${options.id}"]) only if the user explicitly asked you to keep the current response open for this result, or a non-interactive automation must return it in the same invocation; subagent_cancel stops it, subagent_check peeks at a running one, subagent_list shows all.` ); diff --git a/tests/extensions/subagents/print-host.test.ts b/tests/extensions/subagents/print-host.test.ts new file mode 100644 index 00000000..8be70f88 --- /dev/null +++ b/tests/extensions/subagents/print-host.test.ts @@ -0,0 +1,37 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { + buildNonInteractiveSpawnContinuation, + buildPrintHostPendingFollowUp, + canDeliverLaterFromHost, + printHostNeedsFollowUp, +} from "../../../extensions/subagents/src/print-host.ts"; + +test("only the interactive TUI can deliver a later parent turn", () => { + assert.equal(canDeliverLaterFromHost({ hasUI: true, mode: "tui" }), true); + assert.equal(canDeliverLaterFromHost({ hasUI: true, mode: "rpc" }), false); + assert.equal(canDeliverLaterFromHost({ hasUI: false, mode: "tui" }), false); + assert.equal(canDeliverLaterFromHost({}), false); +}); + +test("print hosts with running direct children need a follow-up barrier", () => { + assert.equal(printHostNeedsFollowUp(false, 1), true); + assert.equal(printHostNeedsFollowUp(true, 1), false); + assert.equal(printHostNeedsFollowUp(false, 0), false); +}); + +test("print-host pending follow-up names the ids and forbids ending the turn", () => { + const text = buildPrintHostPendingFollowUp(["sa-1", "sa-2"]); + assert.match(text, /sa-1/); + assert.match(text, /sa-2/); + assert.match(text, /MUST call subagent_wait/); + assert.match(text, /no automatic re-invoke/i); +}); + +test("non-interactive spawn continuation requires wait and does not release the turn", () => { + const text = buildNonInteractiveSpawnContinuation("sa-9"); + assert.match(text, /subagent_wait\(ids: \["sa-9"\]\)/); + assert.match(text, /MUST call subagent_wait/); + assert.doesNotMatch(text, /end your turn/i); + assert.doesNotMatch(text, /automatically re-invoked/i); +}); diff --git a/tests/extensions/subagents/prompt.test.ts b/tests/extensions/subagents/prompt.test.ts index 1b6e0fd3..53736b9b 100644 --- a/tests/extensions/subagents/prompt.test.ts +++ b/tests/extensions/subagents/prompt.test.ts @@ -4,23 +4,23 @@ import assert from "node:assert/strict"; import { readFile } from "node:fs/promises"; import test from "node:test"; import { Value } from "typebox/value"; +import { + AGENT_TYPE_LIMITS, + type AgentType, + BUILT_IN_AGENT_TYPES, +} from "../../../extensions/subagents/src/agent-types.ts"; import { MAX_RUNNING } from "../../../extensions/subagents/src/manager.ts"; import { buildAgentTypeParameterDescription, buildSubagentSpawnResult, - createSubagentSpawnToolSurface, createAgentTypeParameterSchema, + createSubagentSpawnToolSurface, SUBAGENT_SCHEMA_BUDGETS, - SUBAGENT_SPAWN_PROMPT_GUIDELINES, SUBAGENT_SPAWN_PARAMETER_DESCRIPTIONS, + SUBAGENT_SPAWN_PROMPT_GUIDELINES, SUBAGENT_SPAWN_TOOL_DESCRIPTION, SUBAGENT_WAIT_TOOL_DESCRIPTION, } from "../../../extensions/subagents/src/prompt.ts"; -import { - AGENT_TYPE_LIMITS, - BUILT_IN_AGENT_TYPES, - type AgentType, -} from "../../../extensions/subagents/src/agent-types.ts"; function spawnSurfaceBytes(agentTypes: readonly AgentType[]) { const surface = createSubagentSpawnToolSurface(agentTypes); @@ -292,6 +292,21 @@ test("interactive spawn guidance releases the turn instead of waiting on depende assert.doesNotMatch(result, /next step truly cannot proceed/i); }); +test("print-host spawn guidance requires wait and does not release the turn", () => { + const result = buildSubagentSpawnResult({ + id: "sa-1", + title: "review", + harness: "pi", + modelLabel: "m", + cwd: "/repo", + canDeliverLater: false, + }); + assert.match(result, /MUST call subagent_wait/); + assert.match(result, /subagent_wait\(ids: \["sa-1"\]\)/); + assert.doesNotMatch(result, /end your turn/i); + assert.doesNotMatch(result, /automatically re-invoked/i); +}); + test("blocking wait is reserved for an explicit synchronous contract", () => { assert.match( SUBAGENT_WAIT_TOOL_DESCRIPTION, From a10103da192b64d9d00e297a361ba9d396e4ec1f Mon Sep 17 00:00:00 2001 From: tt-a1i Date: Wed, 9 Sep 2026 02:01:00 +0800 Subject: [PATCH 2/2] docs(research): record #512 version split and evidence-gap close Correct E-06..10 as pre-fix, point at bun/unit logs and hang issue #516, and keep the reliability notes from being read as a Benchmark. --- ...LIABILITY_E512_VERSION_SPLIT_2026-09-09.md | 18 +++++++++++ .../CURSOR_RELIABILITY_PROGRESS_2026-09-08.md | 31 ++++++++++--------- .../CURSOR_RELIABILITY_SUMMARY_2026-09-08.md | 22 ++++++------- docs/research/README.md | 2 ++ 4 files changed, 48 insertions(+), 25 deletions(-) create mode 100644 docs/research/CURSOR_RELIABILITY_E512_VERSION_SPLIT_2026-09-09.md diff --git a/docs/research/CURSOR_RELIABILITY_E512_VERSION_SPLIT_2026-09-09.md b/docs/research/CURSOR_RELIABILITY_E512_VERSION_SPLIT_2026-09-09.md new file mode 100644 index 00000000..a3a41aff --- /dev/null +++ b/docs/research/CURSOR_RELIABILITY_E512_VERSION_SPLIT_2026-09-09.md @@ -0,0 +1,18 @@ +--- +status: draft +created: 2026-09-09 +last-verified: 2026-09-09 +applies-to: isolated E high-fast receipts; exploratory, not a Benchmark +related-issues: "#512" +related-prs: "#515" +--- + +Canonical machine+human index lives outside git: + +- `/tmp/openpi-reliability-20260908/results/e-high-fast-512-version-split.md` +- `/tmp/openpi-reliability-20260908/results/e-high-fast-512-version-split.json` +- `/tmp/openpi-reliability-20260908/results/source-overlay.json` + +**E-01..10** = pre-fix stock spawn / `a3edee28` behavior (7/10). E-06..10 are in this window. + +**E-11..20** = post-fix spawn text from commit `7950570` (10/10). Receipt `source` fields on those rows are still the hardcoded stock SHA; do not treat that field as product HEAD. diff --git a/docs/research/CURSOR_RELIABILITY_PROGRESS_2026-09-08.md b/docs/research/CURSOR_RELIABILITY_PROGRESS_2026-09-08.md index ed412d75..71ffa496 100644 --- a/docs/research/CURSOR_RELIABILITY_PROGRESS_2026-09-08.md +++ b/docs/research/CURSOR_RELIABILITY_PROGRESS_2026-09-08.md @@ -1,10 +1,10 @@ --- status: draft created: 2026-09-08 -last-verified: 2026-09-08 +last-verified: 2026-09-09 applies-to: isolated v0.8.1 worktree a3edee28 plus print-host wiring and unused-unless-set OPENPI_CURSOR_WIRE_LOG; exploratory, not a Benchmark -related-issues: "#234, #512, #513, #514" -related-prs: "#484" +related-issues: "#234, #512, #513, #514, #516" +related-prs: "#484, #515" supersedes: none --- @@ -16,11 +16,11 @@ Companion to [`CURSOR_RELIABILITY_BASELINE_2026-09-08.md`](CURSOR_RELIABILITY_BA | Config | A | B | C | D | E | F | | --- | --- | --- | --- | --- | --- | --- | -| cursor-grok-4.6-high-fast | 5 (3 correct, 2 identity-loop timeout) | 5/5 | 5/5 native≥1 | 5 (4 correct, 1 error-retry timeout) | first 5: 2 correct, 3 print-settle; later 15 after-fix all correct | 5/5 | -| cursor-grok-4.6-high | 5/5 | 5/5 | 5 (3 correct+native, 1 wrong+native, 1 no-native) | 5 (4 correct, 1 timeout) | 5/5 | 5/5 (+5 extra correct) | -| cursor-grok-4.6-medium-fast | 5 (4 correct, 1 hang-after-success timeout; acceptor still green) | 5 (4 correct, 1 hang-after-success timeout; acceptor still green) | 5 (4 correct+native, 1 timeout no-native) | 5 (2 correct, 3 timeout) | 5/5 | 5/5 | +| cursor-grok-4.6-high-fast | 5 (3 correct, 2 identity-loop timeout) | 5/5 | 5/5 native≥1 | 5 (4 correct, 1 error-retry timeout) | first 5: 2 correct, 3 print-settle. E-06..10 still stock spawn (5/5 correct). E-11..20 print-host after-fix (10/10) | 5/5 | +| cursor-grok-4.6-high | 5/5 | 5/5 | first 5: 4 native-occurring (3 correct, 1 wrong); C-03 no-native. Extra C-06 native+correct → 5 counting | 5 (4 correct, 1 timeout) | 5/5 | 5/5 (+5 extra correct) | +| cursor-grok-4.6-medium-fast | 5 (4 correct, 1 hang-after-success timeout; acceptor still green) | 5 (4 correct, 1 hang-after-success timeout; acceptor still green) | first 5: 4 native-occurring (C-01 timeout, no native). Extra C-06 native+correct → 5 counting | 5 (2 correct, 3 timeout) | 5/5 | 5/5 | -Class C counts only when `nativeRejections ≥ 1`. high-fast C is 5/5 valid recoveries (PR #484 live revalidation, one config, n=5). +Class C counts only when a native request occurred (`native_rejection` on the wire). high-fast C is 5/5. high and medium-fast each now have ≥5 native-occurring runs after extras (old no-native rows kept). ## Long tasks (≥10 min wall + independent external acceptor) @@ -39,7 +39,7 @@ Content-hash long tasks often mismatch because models include an extra `.ts` fil ## Criterion 3 — cancel / disconnect / timeout / cleanup -- Unit: `pi-backend-lifecycle` 20/20 (cancel, quiet thinking >45s, bounded cleanup). `cursor.test` 30/30 (abort, EOF, server cancel ≠ caller cancel, native reject then Pi roundtrip). +- Unit (re-run 2026-09-09, logs captured): `results/fault/pi-backend-lifecycle.unit.log` 20 pass / exit 0; `results/fault/cursor-provider.unit.log` 30 pass / exit 0. - Live cancel `cancel-live-1788887430222`: `sawTool=true`, SIGTERM after first `tool_execution_start`, leftover `[]`, user Sessions 939, class `cancel`. - Live disconnect `disconnect-live-1788887477808`: isolated CONNECT proxy dropped after first tool; next stream `stopReason: error` / `Client network socket disconnected before secure TLS connection was established`; then `agent_settled`; leftover `[]`; Sessions 939. Class `infrastructure failure`. - After harness SIGTERM loops: no leftover `run-cell` processes. Isolation leak from an earlier un-prefixed lifecycle test was removed; Sessions restored to 939. @@ -47,9 +47,9 @@ Content-hash long tasks often mismatch because models include an extra `.ts` fil ## Failure classes (all attempts kept) 1. **Tool-identity loop** (A high-fast 01, 04) — [#513](https://github.com/openpi-dev/openpi/issues/513). Many successful Pi `read`/`rg`; model claims native / `mcp_openpi_*` names; never writes ACCEPT_A. Under investigation. Not connection timeout. -2. **Hang after acceptor success** (A/B medium-fast) — work written, session does not exit; harness timeout. Hung-task reliability. +2. **Hang after acceptor success** (A-04, B-02 medium-fast) — [#516](https://github.com/openpi-dev/openpi/issues/516). Work written, session does not exit; harness timeout. Under investigation. 3. **Error-recovery loop** (D timeouts) — repeated failed missing-path reads; never ACCEPT_D. Model strategy. -4. **Print-mode parent settle** (E high-fast 01–03) — [#512](https://github.com/openpi-dev/openpi/issues/512). Spawn then `stop` without wait; `--print` emits `agent_settled`. **Fixed in isolated worktree** (see below). +4. **Print-mode parent settle** (E high-fast 01–03) — [#512](https://github.com/openpi-dev/openpi/issues/512). Spawn then `stop` without wait; `--print` emits `agent_settled`. Wiring on research branch / [PR #515](https://github.com/openpi-dev/openpi/pull/515) (not merged). Version-split index: `/tmp/openpi-reliability-20260908/results/e-high-fast-512-version-split.md`. 5. **Native reject without task recovery** (C high 02) — rejection recorded, zero Pi tools, wrong result. 6. **Insufficient evidence** (C high 03; first live cancel) — no tools / no native frames. 7. **Long-task hash canonicalization** — systematic extra-file or digest drift on content-hash shards. @@ -57,19 +57,22 @@ Content-hash long tasks often mismatch because models include an extra `.ts` fil ## Print-host fix (#512) -Before (stock v0.8.1 spawn text), same E task+acceptor, `cursor-grok-4.6-high-fast`, n=10: **7/10** correct, **3/10** print-settle wrong result. +Before (stock v0.8.1 spawn text), same E task+acceptor, `cursor-grok-4.6-high-fast`, **E-01..10** (not E-06..20): **7/10** correct. E-06..10 are still pre-fix. -After (print-host spawn text + bounded `followUp` barrier), same task+acceptor, n=10: **10/10** correct. E-11 spawn result: `This host cannot re-invoke you after you end the turn. You MUST call subagent_wait...` then the model called Pi `subagent_wait` (after a brief `mcp_openpi_subagent_wait` thought — #513 residue). +After (print-host spawn text + bounded `followUp` barrier), **E-11..20**: **10/10** correct. E-11 spawn result: `This host cannot re-invoke you after you end the turn. You MUST call subagent_wait...` -`bun run check` and `bun run test` in the isolated worktree: pass (1470 node tests + 113 vitest; 1 skipped). +Receipt `source` fields remain `a3edee28` on those 20 rows (harness hardcoded). Overlay: `results/source-overlay.json`. New C extras record HEAD `7950570`. + +Isolated `bun run check` exit 0 (`results/isolated-bun-check.log`) and `bun run test` exit 0, 1470 pass (`results/isolated-bun-test.log`). No tools disabled, permissions unchanged, no timeout lengthening, no swallowed errors. ## Issues -- [#512](https://github.com/openpi-dev/openpi/issues/512) print-mode settle — evidenced; fix in isolated worktree, not merged. +- [#512](https://github.com/openpi-dev/openpi/issues/512) print-mode settle — evidenced; wiring on [PR #515](https://github.com/openpi-dev/openpi/pull/515), not merged. - [#513](https://github.com/openpi-dev/openpi/issues/513) Cursor prompt vs advertised names — under investigation; no 10/10 A-loop before-after yet (only 2 failing / 5 on high-fast A). - [#514](https://github.com/openpi-dev/openpi/issues/514) explorer inherit write — under investigation; no proven write escape. +- [#516](https://github.com/openpi-dev/openpi/issues/516) hang-after-success on print after green acceptor — under investigation. ## Sample limits diff --git a/docs/research/CURSOR_RELIABILITY_SUMMARY_2026-09-08.md b/docs/research/CURSOR_RELIABILITY_SUMMARY_2026-09-08.md index d6f8fad4..9e9c68ce 100644 --- a/docs/research/CURSOR_RELIABILITY_SUMMARY_2026-09-08.md +++ b/docs/research/CURSOR_RELIABILITY_SUMMARY_2026-09-08.md @@ -1,10 +1,10 @@ --- status: draft created: 2026-09-08 -last-verified: 2026-09-08 +last-verified: 2026-09-09 applies-to: isolated OpenPI v0.8.1 experiments; exploratory, not a Benchmark -related-issues: "#512, #513, #514" -related-prs: "#484" +related-issues: "#512, #513, #514, #516" +related-prs: "#484, #515" supersedes: none --- @@ -16,26 +16,26 @@ Exploratory. Not a formal Benchmark. Full matrix and classifications: [`CURSOR_R - **Runtime:** Pi 0.85.1; experiment source v0.8.1 `a3edee28`; unique isolated package `/tmp/openpi-reliability-20260908/src`. Daily OpenPI remains `openpi-main-runtime` @ `ed9dbc10` (0.7.0) — unused. - **Configs:** `cursor-grok-4.6-high`, `cursor-grok-4.6-high-fast`, `cursor-grok-4.6-medium-fast` all listed and all streamed live. No silent substitution. -- **18 baseline cells × 5** on the three configs (A–F). high-fast C is 5/5 native-reject → Pi recover (PR #484). +- **18 baseline cells × 5** on the three configs (A–F). Class C native-occurring counts: high-fast **5/5**; high **4/5 then +C-06 → 5**; medium-fast **4/5 then +C-06 → 5**. Old no-native rows kept. - **Six 10+ minute tasks** with external acceptors (LP-high-fast-01 path-hash pass at 789s; five content-hash runs 633–1710s with evaluated wrong-result acceptors). -- **Cancel / disconnect / timeout / cleanup:** unit 20+30; live cancel with tool-seen; live proxy-drop TLS disconnect; leftover processes empty; user Sessions 939. -- **#512 fix:** E high-fast 7/10 → 10/10 on the same acceptor. Isolated `bun run check` + `bun run test` green. +- **Cancel / disconnect / timeout / cleanup:** unit logs recaptured (`results/fault/*.unit.log`, 20/20 and 30/30, exit 0); live cancel/disconnect unchanged; leftover processes empty; user Sessions 939. +- **#512:** E-01..10 stock spawn 7/10; E-11..20 print-host 10/10. Version-split index in `results/e-high-fast-512-version-split.md`. Isolated bun logs: `results/isolated-bun-check.log` / `isolated-bun-test.log` (both exit 0). PR #515 not merged. ## What remains unreliable - high-fast class A identity loop (~2/5). [#513](https://github.com/openpi-dev/openpi/issues/513). No 10/10 before-after yet. -- medium-fast hang-after-success (acceptor green, process does not exit). +- medium-fast hang-after-success (acceptor green, process does not exit). [#516](https://github.com/openpi-dev/openpi/issues/516). - D error-retry loops (especially medium-fast 3/5 timeout). -- C on high: 3/5 valid native recoveries. +- C on high: 3 correct + 1 wrong among 5 native-occurring (plus earlier no-native C-03). - Long content-hash acceptors (model canonicalization). - Explorer/reviewer inherit write/bash. [#514](https://github.com/openpi-dev/openpi/issues/514). ## Before / after (#512 only) -| Source | Task | n | Correct | +| Source | Attempts | n | Correct | | --- | --- | --- | --- | -| v0.8.1 stock spawn text | E / high-fast | 10 | 7 | -| v0.8.1 + print-host wiring | E / high-fast | 10 | 10 | +| v0.8.1 stock spawn (E-01..10; E-06..10 still this window) | E / high-fast | 10 | 7 | +| print-host `7950570` spawn text (E-11..20) | E / high-fast | 10 | 10 | Other cells were not re-run after the fix except in-flight longs. diff --git a/docs/research/README.md b/docs/research/README.md index 7484315e..402b48cc 100644 --- a/docs/research/README.md +++ b/docs/research/README.md @@ -16,6 +16,8 @@ Research records preserve sourced investigation and distinguish observations, in - [`CURSOR_RELIABILITY_SUMMARY_2026-09-08.md`](CURSOR_RELIABILITY_SUMMARY_2026-09-08.md) — configs/tasks verified, remaining unreliability, before/after deltas, sample limits (exploratory; not a Benchmark). +- [`CURSOR_RELIABILITY_E512_VERSION_SPLIT_2026-09-09.md`](CURSOR_RELIABILITY_E512_VERSION_SPLIT_2026-09-09.md) — pointer to the E-01..10 vs E-11..20 overlay (exploratory; not a Benchmark). + - [`WEB_STARTUP_2026-09-07.md`](WEB_STARTUP_2026-09-07.md) — terminal startup feedback, browser-launch waiting, and exploratory timing limits ([#450](https://github.com/openpi-dev/openpi/issues/450)). - [`WEB_STREAMING_MARKDOWN_2026-09-07.md`](WEB_STREAMING_MARKDOWN_2026-09-07.md) — unchanged historical Markdown parsing during streaming, bounded React component reuse and deterministic regression evidence ([#434](https://github.com/openpi-dev/openpi/issues/434)).