feat(atb2): merge_issue, the PR watcher; package written against canary's BAML - #4715
Draft
ATX24 wants to merge 12 commits into
Draft
feat(atb2): merge_issue, the PR watcher; package written against canary's BAML#4715ATX24 wants to merge 12 commits into
ATX24 wants to merge 12 commits into
Conversation
…d env, repro pre-check on canary The infrastructure the fix stage runs on, with no agent in it yet: - models: `InProgress.pr` (the draft PR the agent opened) and `Issue.design_doc` (Hard issues); every Issue constructor carries it. - gauge_issues.baml -> gauge_issue.baml (one issue in, one issue out, like create_issue / organize_issue). - ~/.atb2 layout: a cached clone fetched each run, a shared CARGO_TARGET_DIR, one worktree + run dir per issue; open_sandbox / close_sandbox (close runs from `defer`, never throws, verifies the cache is untouched). - sandbox_env REPLACES the child environment: an allowlist, so the agent never sees GH_TOKEN / FEEDBACK_SUPABASE_KEY / ANTHROPIC_API_KEY, no ambient git credentials or ssh agent, host git config neutralized. Every child runs under `caffeinate -i` with a timeout. - The repro pre-check: repros are re-run out of process on canary HEAD's freshly built baml-cli before a sandbox is opened (a compiler crash is "still broken", not a dead pipeline); repro paths are collapsed so a dataset/LLM-supplied name cannot escape the scratch dir. - Token-free unit tests for the pure parts (naming, env allowlist, path sanitising, verdicts, budgets).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
⏭️ Performance benchmarks were skippedPerf benchmarks (CodSpeed) are opt-in on pull requests — they no longer run on every push. They always run automatically after merge to To run them on this PR, do any of the following, then push a commit (or re-run CI):
|
…guards the environment only; handle_issue tests join the default run - exit 126/127 means the CLI did not start (run_with could not launch it, or caffeinate could not exec it); it read as Fixed for a should-not-compile repro without a diagnostic, and still_broken dropped the issue - the sandbox_env comment claimed more than it delivers: HOME is the host's so the Claude Code CLI can log in, which leaves ~/.ssh and ~/.config/gh readable to the agent - setup_database.sh's default selection runs root::handle_issue::* too
ATX24
force-pushed
the
baml/feedback-part-4
branch
from
September 2, 2026 15:59
3ecc79c to
b457cd3
Compare
ATX24
force-pushed
the
baml/feedback-part-3d-evals
branch
from
September 2, 2026 16:00
bc741e4 to
214b454
Compare
The released toolchain (0.17.0, and the 0.17.1 nightlies) no longer compiles what canary's compiler wants, and the other way round. Move to canary: ctx.output_format() is a call; reflection is its own package (reflect.Package, not baml.reflect.Package); throws unknown on a function declaration is inferred and gone; #"..."# raw strings are backtick strings; baml.sys.ProcessOptions lost keep_stdin_open. setup_database.sh runs canary's own baml-cli (built into ~/.atb2/target by handle_issue; BAML_CLI overrides) instead of pinning BAML_VERSION.
ATX24
force-pushed
the
baml/feedback-part-4
branch
from
September 2, 2026 16:16
b457cd3 to
e3a875e
Compare
ATX24
force-pushed
the
baml/feedback-part-3d-evals
branch
from
September 2, 2026 16:16
214b454 to
5e61fcb
Compare
…rse and Fn@render_prompt The FunctionSpec redesign (#4623) spells the synthesized companions with @ instead of $; the public shape is Fn@spec(args) with .parse() and .prompt(), and the bare companions still compile and run.
…ta, the baml corpus; judged by exit code The pipeline re-runs the repo's gate itself after the agent finishes; the agent's own claim is never trusted. Order and commands follow baml_language/TEST_INSTRUCTIONS.md: fmt --check, clippy -D warnings, `cargo test --lib` for each crate the agent touched, nextest on baml_tests with --no-fail-fast, insta --accept (gating on snapshot state, since verdicts were already taken by nextest), a tree-clean check, then baml-cli built and run over the BAML corpus. Known-flaky nextest failures (ATB2_FLAKY_TESTS, default the one test canary itself fails) are subtracted — but only when nextest exited 100 (compiled, ran, some tests failed). A compile error or a timeout exits otherwise and cannot be waved through on test-controllable output.
…box; PR body; outcome; agents run on Fable The stage itself. For an Easy/Medium issue: a read-only design pass investigates and writes the plan; a fix pass implements it, encodes the repros as tests (routed per TEST_INSTRUCTIONS.md from the repros' expectations) and runs targeted checks; the pipeline re-runs the gate, writes the PR body in the team's shape (Problem / After this PR / Behavioral boundaries / Tests / Validation) and, in Live mode, lifts the worktree's push guard for the one push and opens (or updates) a draft PR. DryRun stops before the push — what the evals run. For a Hard issue the design pass's output becomes `Issue.design_doc` for the shepherd; no code change. The Claude Code CLI is run directly (`claude -p ... --output-format stream-json`), writing its own transcript so the record survives a kill at the budget; implement_fix / write_design_doc are LLM functions only so their $render_prompt / $parse companions render and parse. Both passes run on claude-fable-5 (ATB2_MODEL overrides); time budgets were recalibrated from the first eval (Easy 3600s, Medium 7200s). Every run leaves ~/.atb2/runs/<branch>/outcome.json: kind (fixed / hard / gate_failed / agent_stopped), why the agent stopped, turns, seconds, the gate and the report.
…//# headers for the playground graph - fix_in_budget (PassRate 0.6): every Easy/Medium reference issue with a time_budget_s, DryRun; the agent must land a fix that passes the gate inside the budget. design_doc (PassRate 0.5): Hard reference issues; the design doc must match the reference at LLM-judged similarity >= 0.7. Both are hours of wall-clock and opt in; ATB2_ISSUES narrows. ReferenceIssue gains time_budget_s and design_doc. - run_tests.sh replaces setup_database.sh: resolves the two Supabase secrets through Infisical (and only those — an injected ANTHROPIC_API_KEY makes the Claude Code CLI drop its login), then asks which stage to test: create / organize / pr. - //# headers on create_issue, organize_issue and gauge_issue so the playground's control-flow graph shows named steps.
…issue tests before the evals
…viewer comments back to handle_issue
PR -> green -> merged. `merge_issue("https://github.com/.../pull/N")`
watches any PR of the repo. Each time it finds (1) failing checks, with
the failed steps' log tails via gh run view, or (2) reviewer comments not
yet addressed (CodeRabbit by default; ATB2_REVIEWERS), it renders them
as a brief and calls handle_issue on the PR's branch with it. Returns a
MergeResult: merged / green (waiting on humans) / closed /
waiting_on_checks / out_of_rounds / dry_run. merge_state.json per branch
records the comment ids already put to the agent.
handle_issue takes an optional `review: Review { branch, feedback }`.
Under review it skips the readiness and repro checks, checks the PR's
branch out from origin (open_branch_sandbox), takes the feedback as the
plan instead of running the design pass, then runs the same fix pass,
gate and push it always does, and posts a PR comment saying what was
addressed (and asks CodeRabbit to resolve). The fix prompt says the plan
may be review feedback and how to treat it.
Token-free tests: check/comment parsing, run id from a check link,
pending vs handled, the brief, the PR as an issue, the brief in the
prompt, the state file, the review comment.
…sh runs its token-free tests too The package moved to canary further down the stack; the two raw strings in merge_issue.baml become backtick strings and root::merge_issue::* joins the pr stage of run_tests.sh, before the agent evals.
…eric A PR's branch is built, tested and edited by an agent on this host, so a PR whose branch lives in another repository is refused (MergeResult fork_refused) unless ATB2_ALLOW_FORKS=1. The actions run id parsed from a check's link is used as a gh argument; only a numeric id is accepted.
…o settle and for a push to land - a round only counts its comments as handled when handle_issue ended fixed; any other outcome stops with round_failed since nothing on the PR changed - no round starts while checks are pending, so one round covers CI failures and comments and never pushes over a running CI - after a push, wait for the head to move and its checks to register; an empty check list right after a push read as green - review bodies count as feedback only when they request changes; CodeRabbit's walkthrough after every push kept the loop going until out_of_rounds - review rounds always open the PR branch from origin, not a stale local branch kept after a failed round - merge state moves to ~/.atb2/merge/<branch>, outside the run dir the sandbox wipes; each round's outcome.json is kept - gh pr view is retried three times; failed-check logs keep the tail of every failed step
ATX24
force-pushed
the
baml/feedback-part-4
branch
from
September 2, 2026 17:32
e3a875e to
ebde0d5
Compare
ATX24
force-pushed
the
baml/feedback-part-3d-evals
branch
from
September 2, 2026 17:32
5e61fcb to
561de1e
Compare
ATX24
force-pushed
the
baml/feedback-part-3d-evals
branch
from
September 2, 2026 17:40
561de1e to
2542835
Compare
ATX24
force-pushed
the
baml/feedback-part-3d-evals
branch
from
September 2, 2026 19:40
2542835 to
aaba490
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 4 of the atb2 pipeline:
merge_issue, the PR watcher that gets a PR from opened to green, plus the move of the package onto canary's BAML.What is in here
merge_issue (PR to green to merged). Give it a PR URL; it polls the PR until it merges or closes. Each time CI fails or the reviewer (CodeRabbit by default;
ATB2_REVIEWERS) leaves comments, it gathers them into a brief and hands the PR's branch tohandle_issue, which runs the same fix pass, gate and push it runs for a fresh issue.handle_issuegains an optionalreviewargument for this: no readiness or repro checks, the PR branch is opened from origin, the brief is the plan, and the round ends with a push and a PR comment that says what was addressed and asks CodeRabbit to resolve its threads.Loop behaviour, after a self-review of the first cut:
outcome.jsonhandle_issue writes; comments count as handled only onfixed, anything else stops withround_failedsince nothing on the PR changedout_of_roundsmax_rounds(default 3),DryRunstops before the push,ATB2_POLL_S/ATB2_MAX_WAIT_Sbound the waitSecurity. PRs from forks are refused unless
ATB2_ALLOW_FORKS=1, since the branch is built, tested and driven by the agent on this host. A check's Actions run id is used as aghargument only when it is numeric.Canary migration. The package is written against canary's compiler (
ctx.output_format()as a call,baml.reflect.*, backtick raw strings, nothrows unknown).run_tests.shuses$ATB2_HOME/target/debug/baml-cli(BAML_CLIoverrides).Verification
baml-cli check,fmt,teston canary's compiler: 59 of 59 token-free tests passpr_snapshotagainst feat(atb2): sandbox for handle_issue: worktree lifecycle, allowlisted env, repro pre-check on canary #4634 (head SHA, 41 checks, 3 comments)merge_issueround has been run yet; that pushes to a real PR and is a separate stepBased on #4637 (3d-evals); the stack is #4634, #4635, #4636, #4637, #4639.