diff --git a/docs/agents/examples/test-coverage.mdx b/docs/agents/examples/test-coverage.mdx new file mode 100644 index 000000000..f06225c25 --- /dev/null +++ b/docs/agents/examples/test-coverage.mdx @@ -0,0 +1,195 @@ +--- +title: "Add high-value regression tests" +description: "Review recent merges and open focused, test-only pull requests for material coverage gaps" +--- + +The test coverage example reviews recently merged GitHub pull requests, finds +one meaningful regression risk with weak test coverage, adds the smallest +useful tests, and opens a focused pull request after those tests pass. + +It does not optimize for a coverage percentage. Cosmetic changes, generated +files, broad snapshots, and behavior-neutral refactors are not reasons to +create a pull request. A run that finds no material gap finishes without +writing to GitHub. + +## How it works + +```mermaid +flowchart LR + Merges[Recent merged
pull requests] + Review[Risk and missing-test
review] + Snapshot[Exact source snapshot
and private baseline] + Tests[Minimal tests and
relevant test commands] + Audit{Test-only
audit passes?} + Pull[Focused pull request] + Report[No-op or blocker report] + + Merges --> Review + Review -->|material gap| Snapshot + Review -->|no safe gap| Report + Snapshot --> Tests + Tests --> Audit + Audit -->|yes| Pull + Audit -->|no| Report +``` + +Every run examines a bounded recent-merge window and selects at most one +coherent gap. The agent downloads the exact default-branch commit it inspected, +keeps a private untouched baseline, follows the repository's own test +instructions, and runs the narrowest relevant validation commands. + +The publishing tool compares the tested snapshot with the baseline. It rejects +production changes, unlisted paths, deletions, symlinks, and oversized files +before it can write to GitHub. + +## Start with the example + + + Clone the complete agent, publishing policy, and policy tests from GitHub. + + +```bash +git clone https://github.com/diggerhq/opencomputer-example-test-coverage.git +cd opencomputer-example-test-coverage +npm install +npm test +npm run typecheck +npm run opencomputer -- login +``` + +## Configure a fixture repository + +Start with a disposable repository containing a recent merged change and a +deliberately missing regression test. Set that fixed destination in +`opencomputer/agents/test-coverage/tools/config.ts`: + +```ts +export const TARGET_REPOSITORY = { + owner: "your-github-owner", + repository: "coverage-agent-fixture", + defaultBranch: "main", +} as const; + +export const PUBLISH_ENABLED = false; +``` + +The destination is code-owned. Pull-request text, prompts, commit messages, and +repository files cannot redirect the publishing tool to another repository. + +Create a short-lived, fine-grained GitHub token restricted to the fixture +repository with **Contents: read and write** and **Pull requests: read and +write**. Store it as a Development secret through the hidden prompt: + +```bash +npm run opencomputer -- secrets set GITHUB_PAT \ + --environment development \ + --agent current +``` + +The declared connection limits credential injection to GitHub API requests. +The token is not placed in agent source, prompts, repository URLs, or the +runtime environment. GitHub archive downloads follow the connection's declared +redirect to `codeload.github.com`; they do not require a second secret. + +## Test in Development + +Watch the source and deploy changes to the remote Development environment: + +```bash +npm run deploy -- --watch +``` + +In another terminal, start an explicit run: + +```bash +npm run session -- \ + "Review recent merged code and add the highest-value missing regression tests." +``` + +With `PUBLISH_ENABLED` set to `false`, the agent can inspect the fixture, edit +its isolated snapshot, run tests, and perform the final audit, but the +publishing tool returns a dry-run result without creating a branch or pull +request. + +Review the selected risk, proposed test paths, and observed command results. +If they are correct, set `PUBLISH_ENABLED` to `true`, let the watch deployment +finish, and start a new session with the same request. + +The published branch is deterministically named +`test/coverage-`. Repeating the run for the same repository head +returns the existing open pull request rather than creating a duplicate. + +After changing the example source, wait for the Development watch deployment +to finish and start a new session. Existing sessions remain pinned to the +deployment with which they started. + +## What a dry run looks like + +A useful no-op still reports what it reviewed. This run ranked three recent +merges, identified an entitlement gate as the highest-risk candidate, and +named two concrete missing regression cases before deciding whether it could +safely edit or publish tests. + + + OpenComputer test coverage agent dry-run report ranking three merged pull requests and identifying missing OAuth completion-path and billing-unavailable regression tests + + +The report is evidence for a human reviewer, not permission to publish. The +agent proceeds only after it materializes the exact repository snapshot, reads +the real test harness, edits test-only paths, and observes the validation +commands succeed. + +## Troubleshoot remote runs + +### GitHub archive redirect is blocked + +Current example source authorizes GitHub's archive redirect from +`api.github.com` to `codeload.github.com`. If a session reports +`502 egress-redirect-blocked` while materializing the repository, update the +example, wait for a new Development deployment, and start a new session. + +### Filesystem capabilities and launch trust boundary + +The example explicitly registers the runtime-provided `bash` and `read` tools +in `opencode.json` and exposes them from the agent render. They let the agent +inspect, edit, and test the materialized repository snapshot. If an older +deployment reports `Unknown tool: bash`, `Tool is not available for this +request: read`, or `No Code Mode tools are available`, update the example, wait +for a new Development deployment, and start a new session. + +These are broad capabilities. Repository files and test scripts are untrusted +code and may execute inside the session runtime, and file reads are not +enforced as repository-only. For the initial launch, use a disposable or +non-production repository, restrict the fine-grained token to that repository, +keep `PUBLISH_ENABLED` set to `false`, and review the dry-run result before +enabling publication. + +The agent prompt limits shell and file-reading work to the exact materialized +snapshot and reserves GitHub operations for audited code-defined tools. Those +instructions are behavioral guardrails, not a sandbox boundary. Do not target a +sensitive or production repository until a constrained repository executor +replaces the general-purpose filesystem tools. + +## Safety boundaries + +- The agent can add or update tests and fixtures, but it cannot publish a + production-code testability refactor. +- It cannot merge, approve, close, or label pull requests or change repository + settings. +- Failing, flaky, or environment-dependent tests are reported without + publishing. +- Repository content and test output are treated as untrusted evidence rather + than instructions. +- The initial launch uses general-purpose filesystem tools inside the session + runtime; repository and token scoping remain required even with prompt + guardrails. +- This example starts from explicit interactive runs. Add a recurring schedule + only after validating the repository-specific workflow and publication + policy. diff --git a/docs/docs.json b/docs/docs.json index 51998d64d..740bbb43a 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -73,7 +73,8 @@ "agents/examples/gtm-engineer", "agents/examples/feature-flag-hygiene", "agents/examples/github-actions-triage", - "agents/examples/pr-review" + "agents/examples/pr-review", + "agents/examples/test-coverage" ] } ] diff --git a/docs/images/agents/test-coverage-dry-run.png b/docs/images/agents/test-coverage-dry-run.png new file mode 100644 index 000000000..53766d329 Binary files /dev/null and b/docs/images/agents/test-coverage-dry-run.png differ