ci: run devtools verify on the sinnix-prime self-hosted runner - #4584
Conversation
Per-PR hosted CI runs only the quick gate, so a green PR check does not mean tests ran. This workflow runs the selected `devtools verify` on the workstation runner, on pull_request and merge_group, and uploads the run receipt. `concurrency: pytest` with cancel-in-progress false serialises against the host's single pytest slot in GitHub rather than in a held runner, and never discards a run already proving a revision. `clean: false` on checkout keeps the runner's gitignored state -- the testmon datafile and the devshell venv -- so a run selects instead of reseeding; the seed step only fills in a datafile the runner does not yet have, and a missing seed reports a full run rather than refusing. Verification: nix develop --command devtools verify --quick (12 gates ok). The job itself cannot run until the runner is registered (sinnix feat/github-runner) and made required. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGi8wLWGR2HYBh8p8fXFXz
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e353603d5a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| fi | ||
|
|
||
| - name: Verify | ||
| run: nix develop --command devtools verify |
There was a problem hiding this comment.
When a pull request changes flake.nix or devtools, this step evaluates and executes that untrusted revision directly as the self-hosted runner user. The read-only GitHub token does not restrict filesystem access, and branch protection does not sandbox feature-branch code, so such a PR can read or modify workstation state and persist code in the ignored .venv retained by clean: false; use an isolated ephemeral runner or require an explicit trust gate before dispatching to this host. This is especially incompatible with the repository's requirement to keep operator archives and other private local state outside public artifacts.
AGENTS.md reference: AGENTS.md:L13-L17
Useful? React with 👍 / 👎.
| # GitHub rather than holding a runner while blocked on the slot, and a run that | ||
| # is already proving a revision is never discarded. | ||
| concurrency: | ||
| group: pytest |
There was a problem hiding this comment.
With one run active and another pending in this repository-wide pytest group, a third pull-request or merge-group event replaces and cancels the existing pending run even though cancel-in-progress is false. For example, an incoming PR synchronization can cancel a queued merge-group check, leaving that merge candidate with a canceled required verify result instead of test evidence; remove this concurrency group or scope it per ref and let the already-mandatory host pueue slot serialize pytest.
AGENTS.md reference: AGENTS.md:L147-L149
Useful? React with 👍 / 👎.
| fi | ||
|
|
||
| - name: Verify | ||
| run: nix develop --command devtools verify |
There was a problem hiding this comment.
When a PR changes pyproject.toml or uv.lock and the devshell's uv sync --frozen fails, the shell hook only prints dependency sync INCOMPLETE and still runs this command from the retained old .venv. Because the installed-package identity remains unchanged, testmon can continue selecting against the old environment and the required check can pass without ever testing the dependency versions declared by the PR; add a CI provisioning step that fails the job, or make the shell hook failure fatal in this workflow.
Useful? React with 👍 / 👎.
| fi | ||
|
|
||
| - name: Verify | ||
| run: nix develop --command devtools verify |
There was a problem hiding this comment.
If this job is manually canceled or reaches its 120-minute timeout while pytest is active, devtools.pytest_slot._queue stops waiting but never cancels the task already submitted to the user pueue daemon. That detached pytest therefore keeps using this persistent workspace after Actions starts another job and checks out a different revision into it, corrupting both the test result and the shared testmon graph; the workflow needs cancellation cleanup that terminates the submitted pueue task before releasing the runner workspace.
AGENTS.md reference: AGENTS.md:L147-L149
Useful? React with 👍 / 👎.
| if [ -f .cache/testmon/testmondata ]; then | ||
| echo "runner already has a datafile; keeping it" | ||
| elif [ -f "$seed" ]; then | ||
| cp "$seed" .cache/testmon/testmondata |
There was a problem hiding this comment.
On the runner's first invocation, this plain file copy occurs before devtools verify acquires the host pytest slot, so a local managed pytest run can be writing the source SQLite datafile at the same time. A page-level copy during that commit can leave the runner with a corrupt graph; verify then reports graph_unusable, and every later seed step keeps the bad destination instead of replacing it. Take the seed through SQLite's backup mechanism or perform and validate the copy while holding the same host slot used by writers.
AGENTS.md reference: AGENTS.md:L141-L149
Useful? React with 👍 / 👎.
| fi | ||
|
|
||
| - name: Verify | ||
| run: nix develop --command devtools verify |
There was a problem hiding this comment.
This invokes the default affected-test mode, and devtools converts a zero-test selection into the blocking pytest_no_tests_selected result. A documentation-only PR can therefore never obtain this required check from an up-to-date graph; likewise, after the PR run advances the persistent graph, a single-PR merge-group tree with identical file contents selects nothing and fails even though the same revision was just proven. Use --all for the required merge-queue check or explicitly provide a non-vacuous fallback for an empty affected selection.
AGENTS.md reference: AGENTS.md:L141-L145
Useful? React with 👍 / 👎.
1 participant
Summary
Adds
.github/workflows/verify.yml: averifyjob onpull_requestandmerge_groupthat runsnix develop --command devtools verifyon theworkstation runner and uploads
.cache/verify/current-run.json.Problem
Per-PR hosted CI runs only the quick gate (
.github/workflows/ci.ymlgates thetestjob behindif: github.event_name != 'pull_request'). A green PR checktherefore does not mean tests ran, and every merge relies on someone verifying
locally. A merge queue cannot replace the publication sweep without a required
check that carries real test evidence.
Solution
The runner is
[self-hosted, sinnix-prime], registered by sinnix branchfeat/github-runner(services.github-runners.polylogue, non-ephemeral,Slice = sinnixd-work.slice, work directory under/realm/state).concurrency: { group: pytest, cancel-in-progress: false }— the host has onepytest slot; serialising in GitHub means a queued run waits there instead of
holding a runner blocked on the slot, and a run already proving a revision is
never discarded.
clean: falseon checkout keeps the runner's gitignored state — the testmondatafile and the devshell venv — so a run selects rather than reseeding. The
seed step only fills in a datafile the runner does not have yet, copying from
the primary checkout; a missing seed reports a full run rather than refusing.
pytestgroup automatically(
devtools/pytest_slot.py), so the job plane's memory ceiling applies wherethe work runs.
Verification
12 gates ok (format, lint, mypy, generated-surfaces, layering, patterns,
doc-commands, schema-versioning, oracle-integrity, consumer-reachability,
timestamp-doctrine, schema-privacy).
The
verifyjob cannot execute until the operator mints the runner token andthe sinnix change is deployed; this PR adds the workflow only.
Making the check required
Not done here. After the runner has reported
verifyat least once:Residual risk
workflow change could read or write host state outside the checkout. Branch
protection on
masteris the control.clean: falsemeans stale untracked files survive between runs.devtools verifyreports an interpreter or package change as a full run, so a poisonedvenv shows up as a slow run, not a false green.
🤖 Generated with Claude Code
https://claude.ai/code/session_01YGi8wLWGR2HYBh8p8fXFXz