feat(v1): grade in an isolated box, with Harbor-native artifacts - #2144
Open
rasdani wants to merge 18 commits into
Open
feat(v1): grade in an isolated box, with Harbor-native artifacts#2144rasdani wants to merge 18 commits into
rasdani wants to merge 18 commits into
Conversation
Grading in the box the agent worked in leaves a seam a policy under RL pressure
will find: an editable test file, tamperable grading state, an artifact that
leaks the answer. This carries only what a task declares into a second box and
grades there.
Two channels, non-overlapping:
- `Trace.info` stays the durable record (`patch`, `verdict`) and never travels.
- `/logs/artifacts/` is transport — Harbor's in-sandbox convention, collected
with no declaration, restored in the grading box at the original path
("no translation", as in Harbor).
`Task.finalize` is the producer hook — it already means "runtime live, agent
done, before scoring mutates anything", which is exactly Harbor's collect-hook
moment, so `[[verifier.collect]]` maps onto it rather than needing a new stage.
The env composes the boxes; no `Task.scoring_runtime`, and no change to the
`Runtime` teardown contract.
Harbor `task.toml` support: `artifacts = [...]` (string and object form,
`exclude` honored) and `[[verifier.collect]]`. Sidecar `service`, `[verifier].user`
and an explicit `[verifier.environment]` image are rejected at load; `destination`
is inert, being host-trial-directory placement that verifiers has no equivalent
for. A failing collect hook fails the rollout, unlike `harbor run` which logs and
continues — here the output is a grading input, not observability.
`agentic-judge` gains `--env.topology isolated|shared`, defaulting to isolated,
and stops overwriting the judge's runtime policy when it has its own box. The
judge's workspace note is topology-specific: told it stands in the agent's
workspace when it stands in a fresh one, it reads an unmodified tree as failure.
One archive per source rather than one combined tar: BusyBox tar (every
alpine-based image) has no `-r` to append, and per-source `exclude` patterns
cannot share a single create either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
rasdani
commented
Jul 27, 2026
rasdani
commented
Jul 27, 2026
Review pass on the module.
Removed, as dead or redundant:
- `Collected`, a wrapper over a list whose `total_bytes` nothing ever called;
`collect` now returns `list[CollectedArtifact]` directly.
- `_SYSTEM_ROOTS`, thirteen paths guarding against an author typing `/usr` as a
source. Sources come from task config, not the agent, and the size cap already
refuses anything image-sized. Only `/` needs rejecting, because `lstrip('/')`
makes it empty and tar then fails confusingly.
- the overlap check in `_normalize`. It was load-bearing when collection built one
combined tar; once `restore` began clearing every root before extracting any,
overlapping sources became duplicated bytes rather than a correctness bug.
- `_over_cap`, a function that only formatted a string, and `MAX_ARTIFACT_FILES`,
which counted what the in-box `wc -c` already caps.
Narrowed the link rule, which was both unsafe to relax and too blunt as written.
Refusing every symlink rejects real content — `/etc` and any repo carrying a
`.venv` — while dereferencing instead (`tar -h`) fails the whole collection on a
single dangling link. An outward link is only a way into the grading box when
another member sits underneath it, since that member is what gets written through
the link. So leaf links are kept, in-root and dangling links are kept, and the
write-through shape is refused.
224 lines from 356, then 260 with the link rule.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`_vet` and `_escapes` defended a shape our own tooling cannot produce. Neither BusyBox nor GNU tar recurses into a symlinked directory, so `tar -c` never emits the write-through pair (`link -> /etc` plus a member at `link/passwd`) that the symlink analysis existed for; and tar strips leading `/` while no real path component is `..`, so the member-name checks were equally unreachable. What remained was a defence against one attacker: an agent that replaces the tar binary in its own box to poison the grading container. That is the only channel between the two boxes, but it is a remote attack, and fifty lines of subtle analysis to close it was a bad trade — the subtlety misfired twice while being written, first refusing every symlink (which would have broken any repo carrying a .venv) and then reading hardlink linknames in the wrong frame. Accepted risk: an agent that tampers with its own tar can place arbitrary files at arbitrary paths in the grading box. Note that extracting into a scratch directory instead would not have avoided the analysis — it closes relative traversal, but an absolute symlink target still redirects the write. 260 -> 198 lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`publish` came from Harbor's vocabulary for `/logs/artifacts/` ("the conventional
agent publish directory"), but nothing in this codebase names a parameter that
way. The dominant convention is `<what>_path` — `output_path`, `session_path`,
`script_path`, `config_path` — and `write_path` mirrors `Runtime.write`, which is
the call the argument drives. `output_path` was unavailable: it already means the
host-side eval output directory.
The failure key follows: `patch_publish_error` -> `patch_write_error`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The type only ever held `(root, archive)`, so a dict keyed by source path says the same thing with no class: it preserves declaration order, makes a duplicate root impossible by construction, and drops the last dataclass from the module. `collect() -> dict[str, bytes]`, `restore(runtime, collected)` unchanged at its call sites. 191 lines, from 356 at the start of the branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`release()` was runtime lifecycle wearing an artifacts hat. It read and wrote `runtime.stopped` — another object's lifecycle state — from a module that has nothing to do with lifecycle, and kept its own `_PENDING` set of unawaited teardowns while `runtimes/base.py` already owns exactly that kind of bookkeeping in `_LIVE`. It now sits next to the machinery it belongs to, as `Runtime.stop_nowait`: `stopped` is set on self rather than poked from outside, `_PENDING` sits beside `_LIVE` with the atexit backstop that covers both, and the `_nowait` suffix says what it is — `stop`, without waiting — instead of inventing a verb. `release` was also already taken in v1, by `RolloutSession.release`. This does touch `runtimes/base.py`, having earlier concluded it need not. That still holds for what it was about: adding a re-entry guard to `stop()` would have made a failed teardown unretryable. Adding a sibling method changes no existing semantics. artifacts.py is down to 168 lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured what the optimisation was buying: an awaited docker teardown is 66ms. For that it cost a public method on the Runtime ABC, a `_PENDING` set of unawaited tasks, and a defused context manager in env code — `pop_all()` to disarm `provision`, then taking ownership by hand — with a window between the two where a cancellation would orphan the box until the atexit backstop ran at process exit. It was also backwards on the resource question. Overlapping the solver's teardown with the judge's startup means an episode can hold two boxes at once; awaiting means it holds one. On a paid runtime that matters more than the latency does, and the latency is not on the throughput path anyway — the concurrency gate wraps the agent run, not teardown. The barrier is unchanged and was never the teardown: `collect()` returning is what gates everything downstream. `runtimes/base.py` is untouched again, byte-identical to main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`_checked` rejected a relative source, a `..` component, and bare `/`. Those are author mistakes in task config, not anything an agent controls, and a capable author does not need the framework second-guessing a path they wrote. What remains is the one line that was doing work: stripping a trailing slash, so `/work` and `/work/` cannot key two entries for the same tree — the source doubles as the dict key and as restore's `rm -rf` target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rasdani
marked this pull request as ready for review
July 27, 2026 04:09
ApprovabilityVerdict: Needs human review Unable to check for correctness in 5564d3f. This PR introduces a significant new feature (isolated grading topology with artifact collection/restoration between sandboxes), adds ~300 lines of new runtime logic, and changes error handling semantics in capture_patch. The scope and behavioral impact warrant human review. You can customize Macroscope's approvability policy. Learn more. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
`AgentConfig.runtime` defaults to `SubprocessConfig` and has no `None` to tell unset from chosen, so with `isolated` as the default topology the subprocess guard rejected every config that pinned only the solver — including the bundled `configs/agentic_judge.toml` and `test_env_id_agentic_judge`, which is how this surfaced. The env raised at construction before any episode ran. The judge now falls back to the solver's runtime policy when it has not pinned a container, which is the same treatment `harness`, `model` and `sampling` already get for unpinned seats in `Env._episode_agents`. It is also the right default on its own terms: the grading box is supposed to boot from the solver's image, which is what makes "only the delta travels" true. A judge that pins its own runtime keeps it under `isolated`; under `shared` the solver's still wins, since there the judge's effective runtime IS the solver's box. The explicit subprocess guard goes with it — unreachable now that the fallback runs first, and the solver-side check already covers both-are-subprocess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four conflicts, all from tooling churn rather than overlapping design — nobody else has been in artifacts.py, the Harbor artifact wiring, or the topology logic. - verifiers/v1/__init__.py, task.py, tasksets/harbor/taskset.py: import ordering. #2146 relocated `configs.task`/`configs.taskset` and #2148's isort re-sorted the package; my artifacts imports had landed on the same lines. Kept both sides. - envs/agentic_judge/env.py: #2147 changed the solver-runtime guard from `ValueError` to `TypeError` for Ruff 0.16, on the same line where I had rewritten the message (the check no longer means "the judge plays in the solver's box"). Took their exception type with my message. Also converted three pydantic mutable defaults to `Field(default_factory=...)` — `Artifact.exclude`, `TaskData.artifacts`, `HarborData.collect`. Not conflicts, but Ruff 0.16 flags them and CI would have failed on the merged result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Harbor permits a relative `source` for the main service, and the existence probe resolves one against the runtime's workdir — but the tar runs `-C /`, so it did not. `solution.txt` probed `$workdir/solution.txt`, passed, then archived `/solution.txt`. Not an error: a different file, collected and graded silently. Reproduced with a decoy at the root, which is what got collected. Resolution happens at collection rather than at parse: the workdir is a runtime property and `TaskData.workdir` can override it, so it isn't known when task.toml is read. Also applies review comments on the docs — drops a Harbor paragraph that repeated the Shortcomings list two lines below it, and settles on "sandbox" over "box" in the artifacts section. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`PurePosixPath` join already does the whole job — an absolute source discards the workdir, a relative one joins onto it, and a trailing slash normalises away — so the helper and its `rstrip` were both redundant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…failure `capture_patch` treated every failure the same — record `patch_error`, score the rollout anyway. Two different things were being conflated: - the box answered and git refused: a stale `index.lock` from a killed agent command, a deleted `.git`, a `base_commit` the agent rewrote away, a disk it filled. The agent's own environment, so it still records and still scores. A run with no patch grades as a run that changed nothing, which is the right reward. - the box never answered: sandbox gone, exec timed out, transport dropped. Nothing the policy did, so it now raises. Scoring it would feed training a zero that says only that our infrastructure failed. Telling them apart needs more than the exception boundary: `DockerRuntime.run` returns `docker exec`'s own non-zero result rather than raising, so a dead container is indistinguishable from broken git by exit code alone. On the failure path only, one `true` probe asks the box whether it is still there. The judge sandbox is no longer provisioned when the solver rollout errored — its scoring never ran either, so grading it spends a second box to reproduce a failure the trace already records. `finalize` tolerates the missing judge trace. Also corrects the `trace.info` documentation: it is not a transport channel, but an agentic judge does receive the whole serialized trace including `info` at `/tmp/trace.json`, so anything left there is visible to the grader. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comment argued only the cost — a second sandbox spent reproducing a known failure. The correctness argument was the one I got wrong twice while reviewing: `episode.ok` follows the failed trace on its own, and `episode_should_retry` classifies off that trace's errors, so the real exception type is what decides whether to retry. Raising an `EnvError` here would add a second, less specific error and bury it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI caught this: test_env_id_agentic_judge failed on one of three Python jobs and passed on the other two — not a version difference, a flaky judge. Under the isolated default it was put in a fresh box, told "the agent's environment is gone", and asked for a criterion that reads "you verified it with real execution". echo-v1 publishes nothing, so there was nothing to execute; two models talked themselves into a verdict and the third stalled without writing one. The fixture is not the real problem. Nothing in research-environments publishes to the box yet — `git grep -l "write_path\|CONVENTION_DIR\|logs/artifacts"` over its environments/ is empty, because `write_path` is new here and unused. So isolated-by-default would have changed grading for every consumer of that repo's judge setups, and the three plain-Harbor tasksets would have broken outright: their hints open with "Reconstruct the agent's change from the box: `git status`, `git diff`, `git log` in /testbed", which under isolation is a pristine checkout that grades every attempt as untouched. So the default returns to `shared` — the topology that has actually been measured — and `isolated` becomes opt-in per taskset, adopted once that taskset puts its evidence somewhere that travels. The seven capture_patch tasksets are nearly there already; the trace record they rely on does travel. The isolated path itself is unchanged and still verified: two sandboxes, solver's torn down first, artifacts restored at their original paths, trace uploaded, judge told it is in a fresh box. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a6b82e3. Configure here.
`git add -A` staged everything untracked, so a captured patch carried files the task image shipped and the agent never opened. On R2E-Gym that is three per box (`datasets`, `install.sh`, `run_tests.sh`), and the resulting patch fails `git apply` in a fresh container of that same image — which is exactly what an isolated grading box is. The isolated-judge path was broken for every taskset whose image ships untracked files. Drop untracked files whose mtime predates the agent's first turn. The margin is not subtle: R2E-Gym's are dated 2025-01, the rollouts run now. Pass the cutoff as an age rather than an absolute timestamp — a remote sandbox runs on its own machine, and only a duration survives the clock skew. The listing has to happen before `add -A`, after which nothing is "other" any more. `ignore=` unstages named paths on top, for a taskset that knows its image better than the mtime rule does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mtime cutoff leans on the image's untracked files being old. R2E-Gym's are 556 days old, so it holds today — but it holds for no image we build ourselves, where everything is minutes old at rollout time. Record the untracked set in `setup`, before the agent runs, and hand it to `capture_patch` as `ignore`. Same host-memory pattern as `resolve_head`'s SHA, and no clock in the answer. Tasksets that don't call it keep the mtime rule. Also round the mtime cutoff backwards rather than truncating it: truncation and the round trip into the box both push it later, and a cutoff past the agent's first write drops real work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two mechanisms for one job, and the heuristic was the weaker half: it needs the image's untracked files to be old, which is true of R2E-Gym's and of no image we build ourselves. `snapshot_untracked` answers the same question from a record taken before the agent ran, with no clock in it. `capture_patch` is back to `git add -A` plus one unstage of `ignore`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hallerite
reviewed
Jul 28, 2026
|
|
||
| ## Grading artifacts | ||
|
|
||
| An environment can grade in a second sandbox rather than the one the agent worked in, so nothing the agent did to its environment can reach the grader. Only what the task declares crosses over. |
Member
There was a problem hiding this comment.
we def need to de-slop this section imo
Member
There was a problem hiding this comment.
before 0.3.0, i will go over all the docs changes and de-slop but yeah, this is a lot
hallerite
reviewed
Jul 28, 2026
|
|
||
| A declared path that is missing at collection time fails the rollout: it was declared because grading needs it, and grading a partial state scores the rollout wrong rather than loudly failing it. The convention dir is exempt — it is collected for every task, and most never write to it. | ||
|
|
||
| The grading sandbox boots from the same image as the agent's, so the repo and its dependencies are already present. Only the agent's delta has to travel, which is why the collection cap (`vf.artifacts.MAX_ARTIFACT_BYTES`) is sized for a patch rather than a tree. |
Member
There was a problem hiding this comment.
these are specific to the agentic judging env and should not be in tasksets.md imo
hallerite
reviewed
Jul 28, 2026
| @@ -1,25 +1,32 @@ | |||
| """agentic-judge: a solver plays the task, a judge verifies it in the same box. | |||
| """agentic-judge: a solver plays the task, a code-executing judge verifies the work. | |||
hallerite
reviewed
Jul 28, 2026
| `judge/<name>` metrics plus a weighted-mean `judge` reward, composed with the | ||
| taskset's own rewards via `[env.score]` (judge-only by default). | ||
|
|
||
| `--env.topology` decides where the judge stands. Under `shared` (the default) it |
Member
There was a problem hiding this comment.
need a different name here than topology. topology to me means which agents exist and what is their relationship, whereas this is only a subset of that
hallerite
reviewed
Jul 28, 2026
| cls, | ||
| solution: vf.Trace, | ||
| config: "JudgeTaskConfig", | ||
| topology: str = "shared", |
Member
There was a problem hiding this comment.
also can we type as a union of literals and define that as a type?
hallerite
reviewed
Jul 28, 2026
| ignored (overwritten with the solver's).""" | ||
| """The judge agent. Under `isolated` it provisions its own box from this policy; | ||
| under `shared` it plays in the solver's box and this policy is ignored.""" | ||
| topology: Literal["shared", "isolated"] = "shared" |
Member
There was a problem hiding this comment.
lol it's a union of literals here
hallerite
reviewed
Jul 28, 2026
| config.judge = config.judge.model_copy( | ||
| update={"runtime": config.solver.runtime} | ||
| ) | ||
| # The judge inherits the solver's runtime policy unless it pins a container of |
hallerite
reviewed
Jul 28, 2026
| await agents.judge.run(judge_task, runtime=box) | ||
| return | ||
|
|
||
| # Collection is the barrier: once it returns, nothing downstream needs the |
hallerite
reviewed
Jul 28, 2026
| """Harbor's `artifacts` and `[[verifier.collect]]` blocks, narrowed to what a | ||
| single-container runtime can honor. | ||
|
|
||
| The convention dir is deliberately not prepended here (Harbor's |
hallerite
reviewed
Jul 28, 2026
| {**(env or {}), "VF_DIFF_BASE": base_commit or "HEAD"}, | ||
| ) | ||
| if result.exit_code != 0: | ||
| # Not every runtime raises when the box is gone — Docker returns `docker |
Member
There was a problem hiding this comment.
unrelated to this PR, but this seems like a smell @mikasenghaas
hallerite
reviewed
Jul 28, 2026
| @@ -0,0 +1,166 @@ | |||
| """Carry a task's declared files out of the agent's box and into a grading box. | |||
Member
There was a problem hiding this comment.
imo this should not be tied to agentic judging, as it is presented here. this is way more general than that and the comment should thus be more general
hallerite
reviewed
Jul 28, 2026
| @@ -0,0 +1,166 @@ | |||
| """Carry a task's declared files out of the agent's box and into a grading box. | |||
|
|
|||
| Grading in the box the agent worked in leaves a seam a policy under RL pressure will | |||
hallerite
reviewed
Jul 28, 2026
hallerite
left a comment
hallerite
left a comment
Member
There was a problem hiding this comment.
left some comments, overall like the direction!
xeophon
reviewed
Jul 28, 2026
| Two deliberate differences from `harbor run`: | ||
|
|
||
| - **A failing collect hook fails the rollout.** Harbor logs it and carries on, because there the output is observability; here it is a grading input, and a silently absent file makes the verifier score a stale state. | ||
| - **`destination` has no effect.** It positions a file in Harbor's host trial directory; verifiers has no trial directory (the trace is the record), and Harbor never lets `destination` affect verifier-side placement. |
|
|
||
| - Switching to a different verifier-phase network policy ([Harbor Docs](https://www.harborframework.com/docs/tasks/network-policy)) | ||
| - Shared & separate verifiers ([Harbor Docs](https://www.harborframework.com/docs/tasks#verifier-environment-shared-vs-separate)) | ||
| - Sidecar services, and the sidecar artifacts and collect hooks that go with them ([Harbor Docs](https://www.harborframework.com/docs/tasks#sidecar-artifacts-and-collect-hooks)) |
|
|
||
| ## Grading artifacts | ||
|
|
||
| An environment can grade in a second sandbox rather than the one the agent worked in, so nothing the agent did to its environment can reach the grader. Only what the task declares crosses over. |
Member
There was a problem hiding this comment.
before 0.3.0, i will go over all the docs changes and de-slop but yeah, this is a lot
| model's call. Exercises the config surface too: a policy-only prompt | ||
| override (the verdict contract is appended regardless) and | ||
| reward-composition weights.""" | ||
| """The agentic judge over the echo taskset (needs docker), on the default |
| wrong and references can be narrower than the task; do not over-index on how a | ||
| reference solves it. Your verdict is what YOU verified by execution.""" | ||
|
|
||
| SHARED_SANDBOX_NOTE = f"""\ |
| Your sandbox is the SAME box the graded agent worked in, in the state the agent | ||
| left it — its edits (and any scoring side effects) are applied. {_RECORD_NOTE}""" | ||
|
|
||
| ISOLATED_SANDBOX_NOTE = f"""\ |
| """One `[[verifier.collect]]` command, run in the agent's box by `finalize`.""" | ||
|
|
||
| command: str | ||
| timeout_sec: float = 60.0 |
Member
There was a problem hiding this comment.
i would bump considerably, the other timeouts in v1 are really high on purpose as well. 60s could be too short for huge files / compilation scripts
| if hook.service != MAIN_SERVICE_NAME: | ||
| raise ValueError( | ||
| f"{task_dir.name}: collect hook targets service {hook.service!r}; " | ||
| "sidecars need a compose-capable runtime" |
Member
There was a problem hiding this comment.
no one knows what sidecars are in this context
| and grading a partial state scores the rollout wrong rather than failing it. The | ||
| implicit convention sweep is exempt — most tasks never write there. | ||
|
|
||
| One archive per source: BusyBox `tar` (every alpine-based image) implements only |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grades in a second sandbox instead of the one the agent worked in, so a policy under RL pressure can't reach the grading machinery. Only what the task declares crosses over.
Supersedes #2067 and unblocks the judge-env port @mikasenghaas asked for there, now that multi-agent has landed (#1939).
Opt-in.
agentic-judgestill defaults toshared. Isolation is worth having only once a taskset publishes its evidence somewhere that travels — see Adoption below.Usage
A SWE task produces its delta in
finalizeand declares nothing — anything written to/logs/artifacts/is collected by convention:Anything outside the convention dir is declared on the task row:
A Harbor
task.tomlis read as-is:Run it:
An env composes the sandboxes itself with the two primitives:
Design
Trace.infois the durable record (patch,verdict). It is not a transport channel — though an agentic judge does receive the whole serialized trace,infoincluded, at/tmp/trace.json./logs/artifacts/is transport: box → host → box, discarded after restore.Task.finalizeis the producer hook — it already means "runtime live, agent done, before scoring mutates anything", which is Harbor's collect-hook moment, so[[verifier.collect]]maps onto it rather than needing a new stage.collect()returning is the barrier; the env owns sandbox topology.The grading sandbox boots from the task's own image, so only the delta travels. No
Task.scoring_runtimehook.runtimes/,rollout.py,agent.pyandenv.pyare untouched.Harbor
artifacts = [...](string and object form, relative sources resolved against the runtime workdir,excludehonored) and[[verifier.collect]]. Two deliberate divergences fromharbor run:destinationis inert; it places files in Harbor's host trial directory, which verifiers has no equivalent forRejected at load: sidecar
service,[verifier].user, explicit[verifier.environment]image.agentic-judge
--env.topology shared|isolated, defaulting toshared— what #2109 measured. Underisolatedthe judge gets its own sandbox and the workspace note changes with it: a judge told it stands in the agent's workspace when it stands in a fresh one reads an unmodified tree as failure.An unpinned judge runtime inherits the solver's, matching how
harness/model/samplingalready resolve for unpinned seats.capture_patchnow distinguishes its two failure modes: the sandbox answering and git refusing (agent's own environment — recordspatch_error, still scores) from the sandbox not answering (raises, since a zero there says only that our infrastructure failed). Telling them apart needs a liveness probe, becauseDockerRuntime.runreturnsdocker exec's non-zero result rather than raising.Adoption
Nothing in
research-environmentspublishes to the sandbox yet —write_pathis new here. Under isolation today:capture_patchtasksets work, because their judge hints point atinfo.patchin the trace record, which travelsgit status,git diff,git login/testbed", which under isolation is a pristine checkoutSo adopt per taskset, once its evidence travels.
Verification
ruff,ty check verifiers, 909 non-e2e tests. Against real docker sandboxes, via scratch scripts (not committed, perAGENTS.md):collect/restore— 11 cases: convention sweep, declared paths,exclude, symlink clobber, strict missing-source, relative-source resolution, over-cap, subprocess refusalrun()composition — 8 cases with the two agent runs stubbed and everything else real: two distinct sandboxes, solver's torn down before the judge's is provisioned, artifact restored at its original path, trace record uploaded, judge sandbox built from the solver's imagecapture_patchattribution — healthy capture, broken.gitrecords and continues, dead sandbox raisesNo eval has been run. The isolated path has never executed with a real model or on Prime. On Prime I'd expect judge-sandbox tunnel reachability to break first: it is remote, and nothing has previously provisioned a second remote sandbox needing the interception server within one episode.
Notes
tarhas no-r, so collection builds one archive per source.restore()refuses the subprocess runtime — extraction writes to absolute paths, which there is the host filesystem.tarin its own sandbox can place arbitrary files in the grading sandbox; our owntar -ccannot produce an escaping archive, since it does not recurse into symlinked directories.MAX_ARTIFACT_BYTESis 32 MB on the same-image/delta-only assumption. feat(v1): support Harbor's separate verifier environments #2067 budgeted 256 MB for full-tree transfer.artifacts.pyis 166 lines.🤖 Generated with Claude Code
Note
Add isolated grading topology to
AgenticJudgeEnvwith artifact transporttopologyfield ('shared'|'isolated') toAgenticJudgeEnvConfig; in'isolated'mode the solver runs in one container, artifacts are collected, and a fresh judge container is provisioned to grade using only those artifacts.collectandrestorefunctions that tar declared artifact sources (plus a convention dir sweep) out of the solver box and extract them into the judge box, enforcing a per-rollout size budget.capture_patchin git.py to write the patch to an in-sandbox file and unstage specified paths before diffing; addssnapshot_untrackedhelper.[[verifier.artifact]]and[[verifier.collect]]entries fromtask.tomlintoTaskData.artifactsand run collect hooks duringfinalize, failing the rollout on hook error.MAX_ARTIFACT_BYTESfails the entire rollout;restoreraisesArtifactErrorif called on a subprocess runtime.Macroscope summarized 5564d3f.
Note
Medium Risk
Touches rollout failure semantics (artifact/collect strictness, patch capture raising on infra failure) and multi-sandbox agentic-judge flow; default
sharedtopology is unchanged but isolated path adds provisioning and restore edge cases.Overview
Isolated grading is supported by new
verifiers.v1.artifacts(collect/restore,Artifact,CONVENTION_DIR/logs/artifacts/): declared paths plus an optional convention-dir sweep are tarred out of the agent box and restored at the same paths in a fresh grading container, with strict failures (ArtifactError) for missing declared sources, size over cap, or subprocess restore.agentic-judgegains--env.topology shared|isolated(defaultshared). Inisolatedmode the solver box is torn down aftervf.collect; the judge provisions a separate box,vf.restore, and gets an updated workspace prompt. Failed solver rollouts skip judge grading; unpinned judge runtime inherits the solver's only undersharedor when still subprocess-default.Task / Harbor / git:
TaskData.artifactsdeclares extra roots; Harbor loadsartifactsand[[verifier.collect]](hooks run inHarborTask.finalize, fail the rollout on hook failure), rejects separate verifier images and sidecars.capture_patchaddswrite_path,ignore,snapshot_untracked, and splits git failures (recordpatch_error) from dead-sandbox failures (SandboxError).Docs cover grading artifacts and Harbor collect behavior; public API exports the new symbols and
ArtifactError.Reviewed by Cursor Bugbot for commit 5564d3f. Bugbot is set up for automated code reviews on this repo. Configure here.