Skip to content

feat(run): preserve sourced harness config - #287

Merged
slowdini merged 1 commit into
devfrom
feat/codebase-harness-config
Aug 23, 2026
Merged

feat(run): preserve sourced harness config#287
slowdini merged 1 commit into
devfrom
feat/codebase-harness-config

Conversation

@slowdini

Copy link
Copy Markdown
Owner

Closes #260
Part of #244

Why

Realistic codebase-backed evals inherit the repository's agent instructions, harness settings, and
project skills. Preserving that configuration is important for fidelity, but a checked-in copy of
the subject or a staged sibling can contaminate both sides of the comparison. Operator-installed
skills and codebase-provided skills also need different remedies, so treating both as one shadow
source obscures what the evaluator should do.

Strategy

  • Treat the harness descriptor as the source of truth for project skill discovery. skills_dir
    remains the only staging destination, while additional_project_skill_dirs declares
    cross-harness compatibility roots such as OpenCode's .claude/skills and .agents/skills.
  • Inventory matching project skills immediately after codebase provisioning, before exclusion or
    staging can replace them. Preserve all codebase config by default and emit distinct
    codebase-sourced findings in schema-v3 plugin-shadow.json.
  • Add the per-eval codebase.exclude_skill_sources escape hatch. It moves every declared project
    skill root aside symmetrically in both comparison arms, including revision mode and --no-stage,
    while leaving CLAUDE.md, AGENTS.md, settings, and plugins in place.
  • Reuse the staging manifest's backup/restore model for generated-slug collisions and excluded
    roots. Generated staging names displace and restore an occupied directory; explicit
    --stage-name values still refuse to clobber one.
  • Keep operator isolation and codebase policy independent. A descriptor's
    isolates_live_sources = true assertion suppresses only operator-environment findings;
    preserved codebase findings remain validity warnings.
  • Carry the effective exclusion value through conditions.json, dispatch.json, every
    run.json, benchmark.json, and promoted BASELINE.md provenance.

Safety and compatibility boundaries

  • The default remains preservation: project instructions and harness configuration are never
    broadly stripped from sourced codebases.
  • Exclusion is descriptor-scoped and symmetric. A rootless BYOH descriptor records the policy but
    performs no filesystem mutation.
  • Cleanup validates that restored roots belong to the resolved descriptor and that backups live in
    runner-managed temporary directories before removing the active skills directory.
  • Readers accept schema-v2 and historical unversioned shadow artifacts; writers emit schema v3
    with an explicit finding class.

Configuration example

{
  "codebase": {
    "url": "https://github.com/slowdini/example-project",
    "ref": "v1.4.0",
    "exclude_skill_sources": true
  }
}

For OpenCode, this excludes .opencode/skills, .claude/skills, and .agents/skills because its
descriptor declares all three roots. Other project configuration remains available to the agent.

Verification

  • cargo fmt --check
  • cargo build
  • cargo test — 911 library tests, 184 CLI tests, 182 run tests, and doc tests
  • cargo clippy --all-targets -- -D warnings
  • git diff --check

Keep project instructions and harness configuration in sourced codebases, while inventorying matching project skills separately from operator-environment shadows.

Add descriptor-declared compatibility roots and an opt-in exclusion policy that moves project skill directories symmetrically. Preserve staging collisions through runner-managed backups, and carry the effective policy through report provenance.
@slowdini
slowdini merged commit 5894b93 into dev Aug 23, 2026
7 checks passed
@slowdini
slowdini deleted the feat/codebase-harness-config branch August 23, 2026 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Harness config shipped by a sourced codebase

1 participant