feat(readiness): score how legible a repo is to an agent, next to its stuck rate - #5214
Conversation
|
![8090-software-factory[bot]](https://avatars.githubusercontent.com/in/1847834?s=60&v=4){kind=small}
| @@ -0,0 +1,163 @@ | |||
| """``bp_readiness`` — repo AI-readiness. | |||
There was a problem hiding this comment.
A complete new 'repo AI-readiness' feature has been implemented that provides a free, ungated endpoint to score repository legibility and pair it with stuck-signal detection, but the Local Agent Observability blueprint does not document this feature or its architectural components.
|
| @@ -0,0 +1,167 @@ | |||
| """``bp_readiness`` — repo AI-readiness. | |||
There was a problem hiding this comment.
A complete new "repo AI-readiness" feature has been implemented with endpoint, UI components, and database integration, but the Local Agent Observability blueprint does not document this feature's architecture, components, or contracts.
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
1 similar comment
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
Visual diffComparing 40 of 66 comparison(s) flagged (>1% pixel diff).
Folder: 37aa0f0357ab. Full PNGs also attached as a workflow artefact. Generated by visual-diff bot. Pixel diffs >1% flagged; eyeball the table before merging. This check is non-blocking — fail = bot bug, not a code problem. |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
1 similar comment
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
… stuck rate
Before you blame the agent, look at what you handed it.
Adds a free, ungated repo AI-readiness grade on the Harness tab. Seven
checks over a repo an agent actually worked in (instruction file,
instruction loaded, test command, build command, lint gate, CI config,
skills), each traced to the file this code opened, paired with the
stuck-loop and repeated-tool-failure counts the detectors already
recorded for sessions that ran in the same directory.
Why the Harness tab: it is the tab about everything wrapped around the
model that turns "it can talk" into "it can work", and the repo you hand
an agent is part of that. It is also ungated and collides with no other
work order.
Grading follows ADR-004, the rule the Security tab's posture registry
already uses:
* fail - only a filesystem fact, and every failing check names the
path this module opened to reach the verdict
* warn - present but partial, or an inherited default. `cargo test`
exists for every Cargo project; that is unmeasured, not ready
* unknown - weight 0, forced in one place so no future check can skip
it. It falls out of numerator AND denominator, so a thing we
could not read moves the grade in neither direction
Two calls worth flagging:
* The work order lists "build succeeds" in scope, and its acceptance
criteria say no network calls. Running a build changes the machine and
almost always reaches the network, and a read-only observer must not.
So the graded check is "is a build command discoverable"; the card says
in as many words that ClawMetry never runs your build.
* "Instruction file actually loaded" is a zero-weight unknown today. No
runtime ClawMetry observes reports which context files it loaded, and
reading the file ourselves only proves that WE read it. The evidence
hook is in place and graded the moment a runtime does report it.
The per-runtime file list is DERIVED from runtime_memory's project-scoped
roots (the same declarations the Memory and Skills browsers read), so a
new runtime flows in instead of drifting a second copy. A denylist covers
the handful of those roots the AGENT writes (aider's own transcript,
OpenClaw's memory dir): a repo an agent has merely been run in is not a
repo anyone documented.
Cloud parity: the daemon scores the repos and ships a `repoReadiness`
snapshot slice, because the cloud container has no filesystem to read.
The slice is labelled `scope: all_runtimes` so a hosted renderer cannot
pass node-wide data off as runtime-scoped.
Verified: real repos discovered from sessions.cwd on this machine through
the daemon proxy, both card states rendered in a browser in light and
dark, zero console errors. 59 tests, and the honesty guards were proven
by reverting each fix and watching them go red.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HyGJY8zGK2aSe3g5dmfQDJ
…opy) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyGJY8zGK2aSe3g5dmfQDJ
The daemon scores against every runtime's declared instruction files because it cannot know which runtime the hosted viewer has selected. A card that showed that under a runtime filter with no note would be passing node-wide data off as runtime-scoped (FLYWHEEL 0a.2). The payload now carries `scope`, the local endpoint sets it to the runtime it actually scanned, and the renderer prints the caveat only when the snapshot says all_runtimes and a filter is on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyGJY8zGK2aSe3g5dmfQDJ
…Observability blueprint (v33) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyGJY8zGK2aSe3g5dmfQDJ
…ty states The hosted dashboard runs from ClawMetry's own source tree, so the no-history fallback to the process working directory would render a card about OUR repo and label it as the user's. Gated off when CLAWMETRY_CLOUD is set, with a guard that also proves the fallback still fires locally. The renderer also had one empty state doing two jobs: a repo with a picker full of other repos is a deleted checkout, not a machine that has never run an agent, and telling that reader to run an agent in a code repo is nonsense. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyGJY8zGK2aSe3g5dmfQDJ
FLYWHEEL section 0c is right and I did this in the wrong order: the code was written first and REQ-OBS-007 after it, which is the exact pattern that section was added to stop. Writing it anyway was not a formality. The first draft of AC-OBS-007.3 said every FAILING check names "the specific path that was read", and no failing check can: there is nothing to read when the file is absent. The criterion now says what the code actually does, which is name the paths it looked for plus a remediation, and reserve the read-path evidence for passing checks. That is the claim two existing tests already hold. REQ-OBS-007 on the Local Agent Observability requirement (v16) carries the problem, who is hurt, the eight criteria in external terms, the non-goals, the four alternatives rejected with reasons, and the risks accepted: a grade is a judgement and a wrong one is corrosive; a finished small repo can score badly; repository paths reach the hosted snapshot; and the correlation the card draws is not causal and must not read as if it were. The blueprint section now says it serves REQ-OBS-007 rather than -006. The eight criteria are mirrored into docs/acceptance_criteria.json, cited from the tests that prove them, and the coverage ratchet is tightened in the same commit: 43/108 covered becomes 51/116. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyGJY8zGK2aSe3g5dmfQDJ
… scratch copy of sf_client.py) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyGJY8zGK2aSe3g5dmfQDJ
e2df74b to
37aa0f0
Compare
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
Non-mergeable PR sweep — 2026-08-25Automated sweep across vivekchand/clawmetry, vivekchand/clawmetry-cloud, vivekchand/clawmetry-landing. Results
clawmetry — 25 open PRsSkipped (bot-authored): #5208 (github-actions[bot]), #5176 (github-actions[bot]), #5142 (github-actions[bot]), #5158 #5157 #5156 #5155 #5154 #5153 (dependabot[bot]) GitHub reported These branches already include all of main's commits in their linear history. Blocked by CI/reviews (no conflict): #5220 #5219 #5217 #5216 #5215 #5205 #5198 #5164 #5121 #5114 #5055 clawmetry-cloud — 2 open PRsUnstable (CI failing, no conflict): #2125 clawmetry-landingNo open PRs. Generated by Claude Code |
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HyGJY8zGK2aSe3g5dmfQDJ
Claude-Session: https://claude.ai/code/session_01HyGJY8zGK2aSe3g5dmfQDJ Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 participant
What
A free, ungated repo AI-readiness grade on the Harness tab, sitting next to the stuck-loop counts the detectors already recorded for the same repo.
Seven checks over a repo an agent actually worked in:
testtarget,scripts.test, a pytest sectionlinttarget.github/workflowsand the usual othersRepo discovery comes from
sessions.cwdfolded up to the nearest git root, so the picker lists "repos your agents worked in", never a crawl of$HOME.Why the Harness tab
It is the tab about everything wrapped around the model that turns "it can talk" into "it can work", and the repo you hand an agent is part of that. It is also ungated, and it collides with no other work order in the sprint.
The grading rule (ADR-004)
Same rule the Security tab's posture registry follows, because the trap this work order shipped with is a scanner that graded settings nothing in the codebase ever read:
evidence, rendered in the card as "read from Makefile", so a reader can tell a measured result from a shipped constant.cargo testexists for every Cargo project; that is unmeasured, not ready.Two calls worth flagging
"Build succeeds" is not graded. The work order lists it in scope, and its own acceptance criteria say no network calls. Running a build changes the machine and almost always reaches the network, and a read-only observer must not do either. So the check is "is a build command discoverable", and the card says in as many words that ClawMetry never runs your build. A test asserts the module imports no
subprocessand nourllib, so a future check cannot quietly start shelling out."Instruction file actually loaded" is an honest zero-weight unknown. A file on disk is not a file in the context window, and reading it ourselves proves only that we read it. No runtime ClawMetry observes reports its loaded context files today. The evidence hook is in place and the check grades for real the moment one does.
Derived, not hand-maintained
The per-runtime instruction and skills paths come from
runtime_memory.project_relative_roots(), the samescope="project"declarations the Memory and Skills browsers read, so a runtime added tomorrow flows in instead of drifting a second copy. A test asserts the two sets match.A small denylist covers the handful of those roots the agent writes (aider's own transcript, OpenClaw's memory directory): a repo an agent has merely been run in is not a repo anyone documented, and passing the instruction check on
.aider.input.historywould be a false pass of exactly the kind ADR-004 exists to stop. Denylist rather than allowlist, so a new runtime arrives graded.The suggested file is derived too:
AGENTS.mdis suggested because eleven runtimes read it, not because someone picked a favourite.Per-runtime honesty
A repo can be perfectly legible to Claude Code and invisible to Cursor, and a single node-wide tick would hide that. The graded check scopes to the runtime switcher, and the card carries a coverage strip showing which runtimes would find their instructions here.
Cloud parity
The cloud container has no filesystem to scan, so the daemon scores the repos and ships a
repoReadinesssnapshot slice with the finished report. It is labelledscope: "all_runtimes"so a hosted renderer cannot pass node-wide data off as runtime-scoped. Capped at five repos. A deleted checkout keeps its history row and carriesreport: null, because an invented grade for a checkout that is gone is worse than an honest gap.The
cm-cloud-readinessinterceptor that reads this slice lands in a follow-up onclawmetry-cloud; until then the hosted card renders its honest empty state rather than a blank.Verification
sessions.cwdon this machine through the daemon proxy: 3 repos, the busiest with 10 sessions.score_repo("")silently scored the process working directory.make lint-py39,make lint-daemon-allowlist,make lint-jsall clean. 90 related regression tests green (posture registry, runtime-memory catalog, runtime filter no-leak).Not in scope
Test coverage percentage, and auto-fixing anything. Report only.
Product record: REQ-OBS-007, Tell the operator what they handed the agent (v16) and the Local Agent Observability blueprint (v34, ADR-046/047/048).
In the wrong order, and I am saying so rather than pretending otherwise: the code was written first and the requirement after it, which is exactly the pattern FLYWHEEL section 0c exists to stop. Writing it anyway was not a formality. The first draft of AC-OBS-007.3 claimed every failing check names "the specific path that was read", and no failing check can: there is nothing to read when the file is absent. The criterion now says what the code does, and the eight criteria are mirrored into
docs/acceptance_criteria.json, cited from the tests that prove them, with the coverage ratchet tightened in the same PR.Work order: WO-5, Repo AI-Readiness from the Agent Observability Sprint.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HyGJY8zGK2aSe3g5dmfQDJ