Skip to content

[Feature]: Exclude frozen Red tests from SWE-bench model_patch #24

Description

@GenTang

痛点与背景 / Problem and Context

Problem

In the red_green flow, the frozen Red test is materialized into the workspace. SWE-bench evaluation currently collects the complete Git diff, so model_patch contains both:

production fix + generated Red test

This may conflict with SWE-bench’s official test_patch and affect evaluation validity.

期望的解决方案 / Proposed Solution

Keep both deliverables in the workspace, but submit only the production fix to SWE-bench:

Workspace:         production fix + frozen Red test
SWE-bench patch:   production fix only
Trace artifacts:   full patch + Red test patch

Requirements

  • Exclude only the exact paths recorded in frozen_test.files.
  • Do not broadly filter tests/**.
  • Apply filtering only to red_green SWE-bench submissions.
  • Keep direct_execute behavior unchanged.
  • Fail closed if the frozen test cannot be safely excluded.
  • Record excluded paths and submission patch SHA in the trace.
  • Add tests covering tracked and newly created Red test files.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions