Harden the user-driven loop with the loop-engineering rules - #3
Draft
drmikecrowe wants to merge 19 commits into
Draft
Harden the user-driven loop with the loop-engineering rules#3drmikecrowe wants to merge 19 commits into
drmikecrowe wants to merge 19 commits into
Conversation
The final gauntlet run and the EVIDENCE draft were the loop's last two self-report surfaces: the author executed the run and narrated its result. Two fresh-context briefs close that, the same maker-and-checker split the spec-intent and adversary agents already apply. The runner executes the entry point once and transcribes a per-layer verdict; it fixes nothing, and a layer with no log is a failed row whatever the exit code says. The scribe drafts EVIDENCE from artifacts alone and holds no Bash, so it can transcribe a number but never produce one; the author's claims enter through FACTS.md as labeled assertions that never upgrade a status. Both required at Tier 3, optional at Tier 2; author-run remains the fallback, recorded in EVIDENCE as a downgrade like the brief path.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
EVIDENCE is model-written and the runner agent only transcribes; nothing non-model attested that the run happened, on this tree, after the last edit. The entry point's exit trap now writes that record on the green and red paths alike, and the exit code distinguishes a failed layer from a violated orchestration contract from a crash. The runner reads the record verbatim and fails a run whose record is absent or mis-bound; the scribe copies the source state from it; the report's consistency check gains a Record line. Loop-engineering VE-9, VE-11, CO-9, CO-10.
The adversary reads potentially hostile repo content with a live tool set, and the spec reviewer reads text the requester controls; neither brief said what to do with a directive found inside it. Now both: an embedded instruction is a finding to report, never an order to follow. Loop-engineering EX-8.
The adversary's 10-call budget was a limit with no consequence: an uncounted or over-budget round could still be averaged into confidence. Now it is a failed round, recorded and rerun. And the last permitted round fixes and re-briefs only the blocking finding, so it cannot be spent on cosmetics while the blocker stands. Loop-engineering CO-4, CO-13.
Every checker in the gauntlet must show a red before its green counts; the entry point that runs them all was the one checker exempt. Two one-off negative controls close that: an absent layer must redden the audit, and a failing layer must redden the exit and the record. Loop-engineering VE-13.
The prose is the mechanism, so a change that alters what the gauntlet accepts ships with the case that fails without it. Loop-engineering DR-4.
Two review rounds failing the same way prove the fix changed nothing; a third round spends budget to learn it again. Compare what failed, not its wording, and escalate to the human with both attempts. Loop-engineering CO-3, read for a human-driven loop.
drmikecrowe
changed the title
A spec is complete only against the kind of request it answers: a bug without a reproduction cannot produce its RED test, a feature without the problem stated apart from the proposal invites building the wrong thing, a UI change without a visual expectation is unfalsifiable. The reviewer classifies first (bug / ui / feature), holds the spec to that kind's required substance, and reports the kind so the author can dispute the classification. Adapted from the ticket-enhancer judge in the reliable-agentic-lab workshop.
Two failure modes from running loops on a schedule: redoing finished work because the task was named again, and silently retrying past a standing escalation only a human can clear. Both are stops, and both are narrated — silence on a wake reads as a hang. Adapted from the reliable-agentic-lab enhancer loop's state check and sticky needs-human label; MAST's unaware-of-done mode.
Post only to a destination the human already made; mark this skill's own comments so a later run never answers itself; never filter by author, which would also drop the human's approval. Adapted from the reliable-agentic-lab enhancer loop's comment marker and never-create-an-issue rules.
The consistency lines and the stagnation compare are set membership and string matching, and a decision computed that way cannot be talked past. Where a script exists, the script's answer is the answer. The stance is the enhancer loop's: ready comes from check_fields.py, stop from check_stop.py, never from the model's own judgment.
Where the request and the codebase do not settle a value, an explicit proposed value the reviewer can veto beats a hole discovered mid-build, and a value read from the code beats one composed. The enhancer doer's investigate-before-you-invent rule, applied to spec drafting.
Construction was one dense bullet plus fallback tables; now it is six steps. The new piece is the output contract: a layer's command is wired in only if it exits nonzero on violation, writes its own log, and emits the number its EVIDENCE row will cite — changed-line coverage must gate and fail, not print a global percentage. A tool that cannot meet the contract goes to the human, never in as a report-only step.
The tooling exchange with the human becomes a structured artifact: one row per layer with the pinned tool, the command, and the number EVIDENCE gets from it. Approving the spec approves the rows not struck; a struck row is UNAVAILABLE, never substituted.
The builder writes the entry point, so the instrument was the one piece of the trust chain no fresh eyes checked. A read-only fresh-context agent now certifies it against the approved table before first use: wiring, fail-closed traits, the output contract, the commissioning reds, and what the certification binds to. It runs nothing and fixes nothing; certification voids on any later edit. EVIDENCE gains a Gauntlet commissioned field.
…ry point Two closures for the instrument-authored-by-the-author gap: the scribe fails the report when the runner's executed command differs from the Entry point field, and the adversary's hunt list opens with the entry point script whenever the diff touches it — every green number downstream rests on it.
A subagent's final response can come back empty, truncated, or mangled — hit in practice on an adversary round. When that was the only copy, the whole round had to be re-run and re-paid. Both reviewing agents now take a report path in their prompt — logs/spec-intent.md and logs/adversary-round-<n>.md under the task directory — write the complete report there, then return the same text as their response. The orchestrator reads the response as usual and recovers from the file when the reply is damaged; the file is the authoritative copy and EVIDENCE cites it either way. Under worktree isolation logs/ sits in the durable root, so the copy also survives the worktree's cleanup. That requires Write in both briefs' tool lists, scoped in the text to that one file: spec-intent stays codebase-blind, and the adversary's write is exempt from its 10-call budget so persisting the report cannot void a round.
4e9e96c gave the two reviewing agents a report file to survive a lost or truncated response. The other three subagents return through the same channel and carry the same exposure, so the contract now covers all five. Each brief takes a report path — non-blocking, unlike the required inputs: given none, the agent notes it and returns the text alone. The runner and verifier gain Write scoped to that one file; the runner's "edit no file" becomes "edit no project file", and the verifier's description drops "read-only" for "writes only its own report file". The evidence scribe already held Write; it now also persists its report-back block, whose consistency-check per-line result lives nowhere else once the response is lost. The bundled-agents section replaces the two-agent paragraph with a path table (logs/spec-intent.md, gauntlet-verifier.md, adversary-round-<n>.md, gauntlet-runner.md, evidence-report.md) and states once that the write is exempt from every call budget, so persisting a report can never void a round.
Seven improvements from an adversarial reviewer's own report on how its brief
performed, generalized past that review's specifics:
- Budget 10 -> 12, with ingestion planned: call 1 captures the diff to a file,
read in bounded pages. A wide diff overflowed the return and cost an
unplanned read of the persisted result before hunting began; the raise pays
for ingestion instead of exempting it.
- What counts as a call is now stated: every tool invocation, persisted-result
reads and retries included; only the report write is exempt. The reviewer
had to guess and counted conservatively.
- Generator sentences get a width test beside the reach test, in SKILL.md step
2 and the adversary's hunt 3: a finding inside the mechanism but outside the
sentence's noun means the noun is too narrow ("a device" misses the data the
device holds), and both the author's enumeration and the reviewer's stop at
the same boundary.
- Hunt 4 asks the inverse question explicitly: does the fixture describe the
machine this runs on? The round's strongest finding — fixtures modeling a
state the deployment never produces — was reached only by luck.
- Findings carry a three-word severity (unrecoverable | recoverable |
nuisance) and the fixture input that would have caught them, so the author
can triage without re-reading triggers and each finding arrives as an
addable test.
- The doubled deliverable is gone, for all five agents: the file is the
deliverable, the response a path-plus-summary receipt. Full-text-and-file
paid the report's tokens twice; the no-path fallback still returns full text.
Kept unchanged, on the same feedback: round briefing withholds the previous
round's fixes and hands the generator plus enumeration — the reviewer
confirmed it rediscovered none of the listed fixes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hardens the user-driven loop with the applicable rules from the loop-engineering design doc. Based directly on ca70137; depends on nothing after it. Seven commits, one concern each.
The two agents (259b13d)
agents/old-coder-gauntlet.md— a fresh-context runner executes the final gauntlet run: four inputs, one entry-point invocation, per-layer verdict in the closed five-status vocabulary. It fixes nothing; a layer with no log is FAILED whatever the exit code or record claims.agents/old-coder-evidence.md— a fresh-context scribe drafts EVIDENCE from artifacts alone, holds no Bash (it can transcribe a number, never produce one), and writes absent evidence as failing rows. Author claims enter viaFACTS.mdas labeled assertions that never upgrade a status.The hardening (VE/CO/EX/DR rule ids from the doc)
Gauntlet construction, specified
agents/old-coder-gauntlet-verifier(EX-3 at the instrument). The builder writes the entry point, so a read-only fresh-context agent certifies it before first use: wiring against the approved table, fail-closed traits, the output contract, the commissioning reds, and what the certification binds to. Voids on any later edit; EVIDENCE gains aGauntlet commissioned:field.Entry point:header, and the adversary's hunt list opens with the entry-point script whenever the diff touches it.Deliberately not adopted
Outer-loop automation (iteration counting, durable locked state, scheduled drift evaluation) — the human is this loop's controller; those rules activate the day the skill runs on a trigger instead of an ask.
What this does not claim
The entry point's own exit remains the gate; both agents report and decide nothing. The scribe is a model: it removes author-correlation only, and EVIDENCE names who ran and who drafted so a reader can weigh the claim.
Supersedes #2.