feat(run): preserve sourced harness config - #287
Merged
Conversation
Keep project instructions and harness configuration in sourced codebases, while inventorying matching project skills separately from operator-environment shadows. Add descriptor-declared compatibility roots and an opt-in exclusion policy that moves project skill directories symmetrically. Preserve staging collisions through runner-managed backups, and carry the effective policy through report provenance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #260
Part of #244
Why
Realistic codebase-backed evals inherit the repository's agent instructions, harness settings, and
project skills. Preserving that configuration is important for fidelity, but a checked-in copy of
the subject or a staged sibling can contaminate both sides of the comparison. Operator-installed
skills and codebase-provided skills also need different remedies, so treating both as one shadow
source obscures what the evaluator should do.
Strategy
skills_dirremains the only staging destination, while
additional_project_skill_dirsdeclarescross-harness compatibility roots such as OpenCode's
.claude/skillsand.agents/skills.staging can replace them. Preserve all codebase config by default and emit distinct
codebase-sourcedfindings in schema-v3plugin-shadow.json.codebase.exclude_skill_sourcesescape hatch. It moves every declared projectskill root aside symmetrically in both comparison arms, including revision mode and
--no-stage,while leaving
CLAUDE.md,AGENTS.md, settings, and plugins in place.roots. Generated staging names displace and restore an occupied directory; explicit
--stage-namevalues still refuse to clobber one.isolates_live_sources = trueassertion suppresses onlyoperator-environmentfindings;preserved codebase findings remain validity warnings.
conditions.json,dispatch.json, everyrun.json,benchmark.json, and promotedBASELINE.mdprovenance.Safety and compatibility boundaries
broadly stripped from sourced codebases.
performs no filesystem mutation.
runner-managed temporary directories before removing the active skills directory.
with an explicit finding class.
Configuration example
{ "codebase": { "url": "https://github.com/slowdini/example-project", "ref": "v1.4.0", "exclude_skill_sources": true } }For OpenCode, this excludes
.opencode/skills,.claude/skills, and.agents/skillsbecause itsdescriptor declares all three roots. Other project configuration remains available to the agent.
Verification
cargo fmt --checkcargo buildcargo test— 911 library tests, 184 CLI tests, 182 run tests, and doc testscargo clippy --all-targets -- -D warningsgit diff --check