Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,5 @@ A protocol is not a result. A passing repository check is not independent valida
Commit only reviewed, credential-free summaries and bounded evidence receipts. Keep large JSONL, logs, Sessions, candidate workspaces, caches, credentials, and private settings outside Git under a stable archive identity.

No formal Benchmark result is published by [`Decision 0001`](../decisions/0001-documentation-and-evidence-governance.md). Protocols, templates, and individual results require separate review.

New dated reports start from [`TEMPLATE.md`](TEMPLATE.md). Add each governed report to this index and retain every metadata field: the repository check proves structure, reachability, and link integrity only—not the truth or independent validation of a result.
25 changes: 25 additions & 0 deletions docs/benchmarks/TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
status: draft
created: YYYY-MM-DD
last-verified: YYYY-MM-DD
applies-to: OpenPI revision or release
related-issues: "#NNN"
related-prs: none
supersedes: none
source-revision: commit SHA
model: provider/model and immutable version when available
thinking-level: exact setting
task-set: stable task identity
verifier: stable verifier identity
sample-size: exact count
isolation: workspace and scheduling boundary
usage-accounting: receipt or measurement method
failure-classification: explicit taxonomy and counts
limitations: known gaps
evidence-reference: retrievable archive identity and receipt
rerun-entry-point: command or runbook
---

# Benchmark title

Record protocol, results, interpretation, and limitations without embedding private or unbounded raw evidence.
1 change: 1 addition & 0 deletions docs/disciplines.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ reachability, not adoption or evidence-validation state.
| OP-10 | Lint warnings fail the validation round | enforced | yes | `bun run lint` |
| OP-11 | TypeScript is checked without emitting files | enforced | yes | `bun run typecheck` |
| OP-12 | Runtime provenance is verified before diagnosis | manual | no | `bun run provenance` |
| OP-13 | Governed research and Benchmark records are well-formed, indexed, and link-valid | enforced | yes | `bun run check:knowledge-contract` |

`manual` rows are intentional: they document a contributor action that cannot
be proved by a repository-only check without changing the installed Pi state.
26 changes: 26 additions & 0 deletions docs/research/CHILD_ACQUISITION_AND_PROGRESS_2026-09-07.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
---
status: validated
created: 2026-09-07
last-verified: 2026-09-07
applies-to: OpenPI child acquisition and progress projection
related-issues: #428
related-prs: #429
supersedes: none
---

# Child acquisition and progress reliability audit

- Status: validated for deterministic defect reproduction; repair acceptance is recorded in the linked PR
Expand Down Expand Up @@ -56,3 +66,19 @@ Small after-values approach timer/JIT noise. The useful result is removal of who
Regression tests exercise production Effect cancellation and the real Git worktree lifecycle, plus byte-pressure error retention and first-line preview behavior. `bun run check` passed. Standard `bun run test` passed 1446 Node tests with one platform skip and 30 Vitest tests. Two independent reviews are clean after closing the interrupted-startup worktree classification gap. Remote CI status is recorded in the PR. No paid model calls are needed to reproduce these deterministic runtime defects.

A separate synthetic Cursor transport probe identified repeated copying while accumulating a large fragmented Connect frame. It is outside this bounded child acquisition/progress repair, as are speculative Graph recomputation and renderer-retention concerns without a completed failure proof. This record is a diagnostic investigation, not a formal throughput Benchmark or an assertion that every possible performance issue was resolved.

## Verified facts

The deterministic tests and bounded probes described above establish the recorded lifecycle and projection observations at the source boundary.

## Inferences

The repair interpretation is limited to the mechanisms reproduced by those tests; timing measurements are not generalized performance claims.

## Recommendations

Keep child ownership, cancellation, and progress projection within their existing Pi lifecycle seams and rerun the linked PR validation when those seams change.

## Unknowns

Provider-specific behavior, installed-runtime acceptance, and unmeasured workload limits remain outside this record.
26 changes: 26 additions & 0 deletions docs/research/CURSOR_CONTEXT_AND_TRANSPORT_2026-09-07.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
---
status: validated
created: 2026-09-07
last-verified: 2026-09-07
applies-to: OpenPI Cursor context, transport, and transcript projections
related-issues: #431
related-prs: #432
supersedes: none
---

# Cursor context and transport follow-up

- Status: validated source investigation and deterministic reproductions; PR validation is recorded in the linked Issue
Expand Down Expand Up @@ -36,3 +46,19 @@ The separate [viewport PR #327](https://github.com/openpi-dev/openpi/pull/327),
## Acceptance limits

No paid provider calls were needed or made for these deterministic tests. Local SDK/transport tests establish the client behavior; upstream protocol evidence is separately attributed. Local `bun run check` passed. The default full test attempt was interrupted during host scheduling delay; the complete discovered suite then passed at Node file concurrency 2 (1455 passed, one platform skip; Vitest 30/30). Two independent reviews found no actionable defects. Remote CI receipts are recorded in the PR. This investigation is not a formal end-to-end performance Benchmark.

## Verified facts

The local protocol and transcript regressions described above are the verified source-boundary facts for this record.

## Inferences

Allocation and cache observations support the scoped repairs but do not establish provider-side latency or billing effects.

## Recommendations

Preserve the bounded transport and cache identity contracts when changing the Cursor adapter or transcript renderer.

## Unknowns

Live provider behavior, exact tokenization, and cold-start performance remain unmeasured here.
26 changes: 26 additions & 0 deletions docs/research/CURSOR_NATIVE_RECOVERY_2026-09-08.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
---
status: validated
created: 2026-09-08
last-verified: 2026-09-08
applies-to: OpenPI Cursor native request recovery
related-issues: #234
related-prs: none
supersedes: none
---

# Cursor native request recovery

- Status: validated at the boundaries below; exploratory acceptance, not a Benchmark.
Expand Down Expand Up @@ -40,3 +50,19 @@ Outbound protocol instrumentation counted the rejection; it did not infer recove
## Limits

This is two exploratory runs on one model, not a reliability or performance measurement. It does not establish long-task, fan-out, or all-model acceptance. Proxy tunnel timeouts and remote connection aborts remain separate transport failures. The fix neither proves nor assumes intentional third-party blocking by Cursor. Repository validation and live isolated acceptance do not mean the change is merged, released, or installed in the user's regular runtime. Credentials and private Session transcripts are not part of the published evidence.

## Verified facts

The deterministic Cursor recovery tests and the two bounded exploratory runs establish the behavior and limits stated above.

## Inferences

The observed recovery path supports the scoped native-request boundary but does not generalize to all models or workloads.

## Recommendations

Keep native requests fail-closed and resume through Pi-advertised tools with explicit cancellation and rejection bounds.

## Unknowns

Third-party service reliability, long-task behavior, and all-model acceptance remain unknown.
26 changes: 26 additions & 0 deletions docs/research/ISSUE_419_SHELL_PATH_RETROSPECTIVE_2026-09-07.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
---
status: validated
created: 2026-09-07
last-verified: 2026-09-07
applies-to: OpenPI headless Session shellPath inheritance
related-issues: #419
related-prs: #423
supersedes: none
---

# Issue #419:headless Session 的 shellPath 继承复盘

- 状态:在源码与回归测试边界验证
Expand Down Expand Up @@ -43,3 +53,19 @@ Headless Session 重新使用 Pi 的 shell 解析和生命周期。当前环境
因此没有宣称真实 Windows WSL stub 的手工验收;回归测试使用跨平台的“不存在
shell 路径”证据验证覆盖边界。Windows 原生命令编码和 Git Bash 安装发现仍由
Pi 0.85.1 负责。

## Verified facts

源码复核与回归测试确认了 headless Session 的 shellPath 保留边界。

## Inferences

该修复解释了复现的配置丢失,但不等同于 Windows 环境的手工验收。

## Recommendations

继续让 Pi 原生工具拥有 shell 生命周期,展示扩展只在 TUI 模式替换展示行为。

## Unknowns

真实 Windows WSL launcher、Git Bash 安装和用户配置组合仍未在本记录中验证。
2 changes: 2 additions & 0 deletions docs/research/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

Research records preserve sourced investigation and distinguish observations, inferences, recommendations, and unknowns. They are not accepted Decisions or proof of runtime behavior by themselves.

New governed records start from [`TEMPLATE.md`](TEMPLATE.md), retain the required metadata, and link back to their source Issue. The checker currently enforces this contract for `docs/research` and `docs/benchmarks`; the architecture, design, and Decision categories remain documented but are not yet scanned by that checker. Only the explicitly enumerated historical records below are exempt from frontmatter; a new record without frontmatter fails the governance check.

## Source-verified designs

- [`WEB_TRAJECTORY_2026-09-07.md`](WEB_TRAJECTORY_2026-09-07.md) — saved Session trajectory, exact-request capture boundary, and bounded browser inspection ([#446](https://github.com/openpi-dev/openpi/issues/446)).
Expand Down
27 changes: 27 additions & 0 deletions docs/research/TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
status: draft
created: YYYY-MM-DD
last-verified: YYYY-MM-DD
applies-to: revision, release, or source boundary
related-issues: "#NNN"
related-prs: none
supersedes: none
---

# Research title

## Verified facts

State sourced observations and their verification boundary.

## Inferences

Separate interpretations from observations.

## Recommendations

Record proposed action without presenting it as an adopted Decision.

## Unknowns

List unresolved questions and evidence that would answer them.
26 changes: 26 additions & 0 deletions docs/research/WEB_STARTUP_2026-09-07.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
---
status: validated
created: 2026-09-07
last-verified: 2026-09-07
applies-to: OpenPI Web startup and browser launch boundaries
related-issues: #450
related-prs: none
supersedes: none
---

# Web startup feedback and browser waiting

- Status: validated for the source observations below; timings are exploratory.
Expand All @@ -19,3 +29,19 @@ These exploratory observations do not establish a cold-start speedup: operating-
The terminal handoff now writes progress immediately. Host readiness and the usable address are printed before invoking the browser opener. The native `execFile` operation has a three-second deadline and accepts cancellation during Host shutdown. An opener exit is described as an open request, not proof that a browser page loaded. Timeout stops only the owned launcher process; this does not promise cleanup of arbitrary descendants created by operating-system launch services.

`tests/web/startup.test.ts` uses an isolated real CLI and a deliberately stalled opener on POSIX to verify that the address appears and serves HTTP before the opener completes, that timeout preserves the serving Host, and that shutdown cancels the pending launch. The process test is explicitly skipped on Windows; pre-cancelled launch and ready-screen tests are portable. Extension tests check progress before spawn. Browser rendering performance and fully cold user-configured startup remain separate investigation scopes.

## Verified facts

The startup and browser-launch tests establish the readiness, timeout, cancellation, and progress boundaries described above.

## Inferences

The measured local timings support the lifecycle repair but do not prove a general cold-start speedup.

## Recommendations

Keep readiness publication independent from browser opener completion and preserve bounded cancellation on shutdown.

## Unknowns

Cold user-configured startup and browser rendering performance remain unmeasured.
26 changes: 26 additions & 0 deletions docs/research/WEB_STREAMING_MARKDOWN_2026-09-07.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
---
status: validated
created: 2026-09-07
last-verified: 2026-09-07
applies-to: OpenPI Web streaming Markdown rendering
related-issues: #434
related-prs: none
supersedes: none
---

# Web streaming Markdown reuse

- Status: validated for the deterministic component regression below; not a timing benchmark.
Expand Down Expand Up @@ -27,3 +37,19 @@ Only Markdown rendering reuse changes. Sanitization, GFM, soft breaks, URL check
This is an invocation-count regression, not a browser timing or memory benchmark. It does not establish an end-to-end speedup, reduce the production bundle, eliminate list reconstruction or avoid parsing a growing live reply. React can legitimately render again after remounting or other lifecycle changes. A future change that gives Markdown additional inputs must retain React's normal prop comparison or explicitly account for those inputs.

Reproduce the focused evidence with `bunx vitest run tests/web/markdown-streaming.spec.ts tests/web/app-render.spec.ts`. Required full checks and browser/CI receipts belong to the linked PR; a package build or merge does not establish installed-runtime acceptance.

## Verified facts

The component regression establishes the reported Markdown invocation counts and update boundaries.

## Inferences

The reuse boundary removes repeated historical parsing in the covered render path but is not an end-to-end browser performance measurement.

## Recommendations

Retain normal React prop comparison and update the regression whenever Markdown receives additional inputs.

## Unknowns

Installed-runtime latency, memory behavior, and provider acceptance remain outside this record.
26 changes: 26 additions & 0 deletions docs/research/WEB_TRAJECTORY_2026-09-07.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
---
status: validated
created: 2026-09-07
last-verified: 2026-09-07
applies-to: OpenPI Web trajectory evidence projection
related-issues: #446, #76
related-prs: none
supersedes: none
---

# Web trajectory evidence boundary

- Status: source-verified design; deterministic implementation validation belongs to the linked PR. Not a performance benchmark or provider acceptance result.
Expand All @@ -20,6 +30,22 @@ The trajectory is a browser-only projection of the already loaded saved records.

Existing Chat's content-based live deduplication and synthesized live timestamps are unsuitable as an audit source. This implementation does not reuse that merger or silently change Chat behavior. It also adds no full-history scan, endpoint, model tool, request capture, persistence, timer or provider implementation.

## Verified facts

The source review establishes the bounded trajectory projection and its evidence omissions.

## Inferences

The projection is suitable for the recorded browser inspection boundary but cannot attest to final provider payloads or complete execution history.

## Recommendations

Keep future request inspection behind a separately bounded native capture with explicit source and sensitivity rules.

## Unknowns

Historical full requests, complete tool durations, and final wire payloads remain unknown.

## Deferred and unknown

A full request inspector needs a separately bounded native capture after mutable handlers, with explicit source identity and handling of sensitive context. Historical full requests cannot be reconstructed reliably from this Web projection. Real model usage and realtime tool duration could be added through Pi's native message/event seams; unknown values must remain unknown. No claim is made about final wire payload, complete context, historical tool duration, or end-to-end performance.
26 changes: 26 additions & 0 deletions docs/research/WORKFLOW_CHILD_FAILURES_2026-09-07.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
---
status: validated
created: 2026-09-07
last-verified: 2026-09-07
applies-to: OpenPI Workflow child lifecycle and capability inheritance
related-issues: #424
related-prs: #426
supersedes: none
---

# Workflow child failures and capability inheritance

- Status: validated for private run inspection and local regression reproduction; live provider acceptance tracked separately below
Expand Down Expand Up @@ -58,3 +68,19 @@ On 2026-09-07, an isolated Pi agent directory reported this repair worktree as i
| Eight concurrent real Workflow `runAgent` sessions | 8/8 successful; each executed one read, completed two model turns, and returned the correct external-file nonce | 12.0 s total |

These are bounded connectivity/lifecycle acceptance runs, not model-quality or throughput Benchmarks. The eight-child smoke invoked the production runner concurrently; selected-cwd DSL propagation is separately covered by execute-level tests. It does not prove long 44-PR audits fit provider rate limits, automatic choice of the right tool, every Cursor model, or the user's older custom role configuration. Cursor does not report complete usage, so zero provider counters are unknown accounting rather than a claim of free execution.

## Verified facts

The local lifecycle, capability, transport, and bounded smoke tests establish the recorded child behavior.

## Inferences

The repairs explain the reproduced startup, tool-boundary, and timeout mechanisms without proving provider-wide reliability.

## Recommendations

Keep Pi as the owner of provider, tool, cwd, trust, and cancellation lifecycle decisions for child sessions.

## Unknowns

Provider rate limits, all-model behavior, long audits, and legacy custom role configurations remain unknown.
26 changes: 26 additions & 0 deletions docs/research/WORKFLOW_DASHBOARD_REFRESH_2026-09-07.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
---
status: validated
created: 2026-09-07
last-verified: 2026-09-07
applies-to: OpenPI Workflow dashboard refresh and history projection
related-issues: #420
related-prs: #422
supersedes: none
---

# Workflow dashboard refresh investigation

- Status: validated at the source and regression-test boundary
Expand Down Expand Up @@ -56,3 +66,19 @@ changes from other processes become visible on reopening the dashboard. This
record does not claim real-terminal acceptance after reload, release publication,
or elimination of every performance bottleneck. Private sessions, transcripts and
raw process samples remain outside the repository.

## Verified facts

The dashboard regression tests establish the bounded refresh and selected-detail hydration behavior described above.

## Inferences

The scoped cache and event-boundary changes address the reproduced spinner workload without proving elimination of all dashboard cost.

## Recommendations

Keep full history scans outside the spinner tick and retain explicit hydration for the selected run.

## Unknowns

Installed-runtime behavior, cross-process reload latency, and unrelated terminal bottlenecks remain unknown.
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -108,9 +108,10 @@
},
"scripts": {
"prepublishOnly": "bun run check && bun run test",
"check": "bun run check:config-contract && bun run check:discipline && bun run check:web && bun run format:check && bun run lint && bun run typecheck",
"check": "bun run check:config-contract && bun run check:discipline && bun run check:knowledge-contract && bun run check:web && bun run format:check && bun run lint && bun run typecheck",
"check:config-contract": "node scripts/check-config-contract.mjs",
"check:discipline": "node scripts/check-discipline-ledger.mjs",
"check:knowledge-contract": "node scripts/check-knowledge-contract.mjs",
"prepare": "node scripts/prepare-effect-tsgo.mjs",
"format": "biome format --write .",
"format:check": "biome format .",
Expand Down
Loading
Loading