feat(v1): Harbor separate verifier environments

[rasdani]

Stacked on #2144 (feat/isolated-grading-artifacts). Review that first; this PR's base is that branch, not main.

Wires up Harbor's [verifier].environment_mode / [verifier.environment], which we currently ignore and hard-reject respectively — so a task that asks to be graded away from the agent is graded inside the box the agent just had root in.

Design and prior art are @xeophon's, from #2067. This is that work rewritten against #2144 rather than merged with it; see Why not merge #2067 at the end.

Why this is a correctness gap, not a nicety

Verified on r2e-gym while testing something else: an agent-planted /testbed/conftest.py with a pytest_runtest_makereport hook forcing passed scores reward 1.0 with no fix applied. It never touches the grading script or the test files, so hiding those does nothing. Pytest's plugin surface alone offers conftest, sitecustomize, ini files and the venv's own entry points — any grader running in the agent's box is reachable, and patching individual paths is whack-a-mole.

What lands

Task.scoring_runtime (task.py, rollout.py) — returns a context manager to score inside, or None for today's behaviour. RolloutRun resolves it in open(), before the box boots, so a task that scores elsewhere but borrowed its runtime or has shared tool servers fails immediately rather than after a full agent run. Harness and task scoring stop being gathered: the harness's metrics describe the agent's session, so they're read off the agent's box before the task's context manager can tear it down.

Runtime primitives — provision_runtime (the start/stop context manager Agent.provision already had inline), stop_confirmed/teardown_confirmed, and read_bounded. Confirmed teardown is the point of the second one: teardown is best-effort because a leaked container is a billing problem, but a caller standing up a grading box needs the first box gone — an agent can leave a background process running past its final turn.

Harbor wiring (tasksets/harbor/taskset.py) — mode resolution via Harbor's own resolve_task_verifier_mode / resolve_effective_verifier_env_config, a VerifierConfig on HarborData, and verifier_runtime_config deriving the grading box from the agent's.

reward.json — Harbor's separate verifiers write it, not reward.txt. Without this every separate-mode task would score 0, which reads as a failed attempt.

Three cases, and why one is rejected

Harbor resolves the verifier environment as [verifier.environment] if declared, else a deep copy of [environment]:

task.toml	Harbor's verifier image	here
environment_mode = "separate", no [verifier.environment]	the task's own image	supported
[verifier.environment] with docker_image	that image	supported
[verifier.environment] without docker_image	built from tests/Dockerfile	rejected at load

The third is rejected because verifiers pulls images and never builds them — the same rule [environment] already follows. The error says to build and push the image and name the ref. ignore_dockerfile remains the one escape hatch and now warns, because falling back to the agent's image runs the verifier somewhere the task never declared.

Only declared artifacts cross over. That's Harbor's own model — Trial._run_separate_verifier uploads artifacts and nothing else, with no full-tree copy anywhere — so #2144's collect/restore and its 32 MB budget carry this unchanged, and #2067's parallel artifact layer and 256 MB budget are not ported.

Smaller decisions worth a look

• [[verifier.collect]] keeps working in separate mode. feat(v1): support Harbor's separate verifier environments #2067 rejects it. It composes fine: the hooks already run in finalize, in the agent's box, producing exactly the files that then travel.
• The verifier's network mode is enforced, not warned about, using the allow/block policy the runtimes already have. allow_internet = false means the grader really has no network — verified below. feat(v1): support Harbor's separate verifier environments #2067 warns instead; I think a declaration that silently doesn't hold is worse than one that occasionally bites.
• Artifacts are restored before tests/ is staged, and /tests is wiped first. An artifact entry pointing into /tests would otherwise hand the agent the grader's own scripts, and a fresh container of the task image can ship a stale /tests.
• A failed tests/ staging now raises instead of leaving test.sh missing and scoring 0.
• A reward-less reward.json object sums. {"correctness": 1.0, "style": 0.25} records two rewards and totals 1.25 — v1's existing keyed-reward convention, and feat(v1): support Harbor's separate verifier environments #2067's behaviour, but it can exceed 1.0. Flagging rather than changing.
• ignore_separate_verifier forces shared grading when a sandbox per task is too expensive. Not a Harbor knob; same family as ignore_timeouts / ignore_dockerfile.

Two semantics reviewers should check

• Provisioning now runs inside _scoring_timeout, which Harbor sets from verifier.timeout_sec (default 600s). A grading box that can't be reached in time raises a scoring timeout; it must never become reward 0. Verified below with a bad image ref.
• This reverses feat(v1): grade in an isolated box, with Harbor-native artifacts #2144's stated position that the env owns sandbox topology and that this hook shouldn't exist. That holds for the agentic judge, which composes seats it already owns. It doesn't extend to a taskset's own programmatic grader: task.score runs inside the rollout, where no env can reach it. The conftest result above is the argument.

Verification

No public dataset exercises this feature. 104,131 cached task.toml files declare zero environment_mode or [verifier.environment]; neither Harbor Hub (80 datasets) nor laude-institute/harbor-datasets (103) ships one. So: local fixtures for each case, driven through the real finalize → scoring_runtime → score path against live Docker containers.

PASS  shared grades in the agent's box
PASS  shared still sees the agent's leftovers
PASS  fresh_copy gets its own box
PASS  declared_image gets its own box
PASS  agent box gone before grading (fresh_copy)
PASS  agent box gone before grading (declared)
PASS  declared artifact travelled (fresh_copy)
PASS  declared artifact travelled (declared)
PASS  undeclared agent file did NOT travel (fresh_copy)
PASS  undeclared agent file did NOT travel (declared)
PASS  stale /tests wiped in the verifier box
PASS  declared verifier image actually used          # python 3.12 vs the task's 3.11
PASS  fresh copy keeps the task image
PASS  allow_internet=false enforced                  # NET=DOWN in the verifier box
PASS  public verifier keeps network
PASS  reward.json read (shared / fresh_copy / declared)
PASS  wrong answer scores 0

Case C rejected at load with the build-and-push message; loads under --taskset.ignore-dockerfile with a warning; --taskset.ignore-separate-verifier returns verifier=None. A bad verifier image ref raises SandboxError rather than scoring 0. _reward_json checked against 14 payload shapes (scalar, map, map-with-reward, empty, string values, bools, list, malformed, missing).

909 non-e2e tests, ruff check, ruff format --check, ty check verifiers all green. Scratch scripts and fixtures live in /tmp and ~/.cache/harbor, not committed (AGENTS.md).

Not verified: Prime. Everything above is Docker. PrimeRuntime.teardown_confirmed is written but unexercised, and Prime provisioning has been timing out independently — which is exactly the path where the scoring-timeout interaction matters most.

Why not merge #2067

A real test merge of the two branches conflicts in 3 files / 9 hunks, with seven semantic collisions underneath: two Artifact classes, TaskData.artifacts vs HarborData.artifacts, two artifact transports, opposite strictness on the convention dir, and contradictory handling of [[verifier.collect]] and [verifier.environment]. #2067 is also 7 commits behind main. Resolving it would have meant deleting most of its diff, so the surviving design was rebuilt on #2144's transport instead.

Open question for @xeophon

DeepSWE v1.1 isn't on Harbor Hub, in the public dataset repo, or in the local cache, so I couldn't read one of its task.tomls. From your two review comments on #2067 — [verifier.environment] present, allow_internet = false, no docker_image, Harbor building tests/Dockerfile — it looks like case C, the one this PR rejects, meaning DeepSWE would need its verifier image built and pushed first. One real task.toml would settle it.

🤖 Generated with Claude Code

---

Note

High Risk
Changes core rollout scoring topology and sandbox lifecycle (confirmed teardown before grading); incorrect isolation or timeout handling could mis-score rollouts or leak agent influence on graders.

Overview
Adds isolated grading for Harbor tasks that set [verifier].environment_mode = "separate": the agent runs in one sandbox, then only declared artifacts move into a second box for test.sh and scoring, after the agent box is confirmed torn down.

Task.scoring_runtime is the extension point (default None = score in the agent box). RolloutRun resolves it before boot, rejects borrowed runtimes or shared MCP when a separate box is required, and runs harness scoring on the agent box before entering the task’s scoring context manager so metrics still reflect the agent session. Verifier provisioning counts against _scoring_timeout.

Runtime helpers: shared provision_runtime (used by Agent.provision), stop_confirmed / teardown_confirmed on Docker and Prime, and read_bounded for capped reads inside the box.

Harbor: stops rejecting [verifier.environment]; parses separate mode via Harbor’s resolvers, VerifierConfig, ignore_separate_verifier, verifier network policy on the grading box, artifact collect/restore + wiped /tests staging, reward.json parsing with reward.txt fallback, and load-time rejection when Harbor would build tests/Dockerfile without a pullable image.

Docs updated for separate verifiers and remaining gaps (shared-verifier network policy, Dockerfile builds).

^{Reviewed by Cursor Bugbot for commit ef5503b. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add separate verifier environments for Harbor task scoring

• Introduces a VerifierConfig dataclass and HarborTask.scoring_runtime override so Harbor tasks can grade in a freshly provisioned, isolated runtime (Docker or Prime) instead of the shared agent box.
• Adds Task.scoring_runtime hook to task.py and scoring path in rollout.py: harness scoring runs on the agent box first, then task scoring runs inside the separate verifier box within the scoring timeout.
• Adds stop_confirmed / teardown_confirmed to base.py with implementations for Docker and Prime runtimes, ensuring the agent box is fully torn down before the verifier box is provisioned.
• Adds read_bounded to Runtime for size-capped file reads, and _reward_json to HarborTask to parse structured reward.json rewards before falling back to reward.txt.
• Adds provision_runtime async context manager in runtimes/init.py for automatic start/stop lifecycle, used in both the refactored agent provisioning and verifier box setup.
• Risk: rollouts that request a separate scorer must own the agent runtime and have no shared tool servers — mismatched configurations raise at init time.

^{Macroscope summarized ef5503b.}

[macroscopeapp]

Approvability

Verdict: Needs human review

Unable to check for correctness in ef5503b. This PR introduces a substantial new feature for running verifiers in isolated sandbox environments, with new runtime lifecycle methods (stop_confirmed, teardown_confirmed), changes to the scoring flow, and integration with Harbor's separate verifier mode. The scope and runtime implications warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[xeophon]

why do we name config policy here?

[xeophon]

doesn't have to do anything with single-container, no?

[xeophon]

what about time? verification also has a time config, which we now ignore by default (+ supports multiplier) but we need to wire it up here