Skip to content

keep_workspaces = false never reaps task worktrees #1227

Description

@mrthankyou

Summary

With keep_workspaces = false, a fully successful run still keeps every task worktree. Smithers logs warn keeping worktree with unsaved work once per task, because each worktree contains files Ultrafuzz itself wrote there. ultrafuzz clean <run> then deletes the run directory but leaves the Git worktree registrations, the per-task branches and the Smithers run record. Relaunching with the same run ID fails at admission after ~7 minutes of preparation.

Evidence (main 6ac8c2bf, 2026-09-29)

Reproduced in two smoke runs against Damn Vulnerable DeFi v4.1.0 (Claude, 8/8 nodes succeeded). The second ran on 6ac8c2bf plus an unrelated agent-failure-reporting change.

Worktrees are never reaped

  • All 8 task worktrees remain under <run>/workspaces/, and the engine log contains warn keeping worktree with unsaved work 8 times.
  • Every worktree's git status shows only Ultrafuzz-owned paths:
    ?? .ultrafuzz/
    ?? artifacts/
    
    Strategy nodes additionally show ?? test/foundry/ and, in one case, ?? foundry.lock.
  • Worktrees were 19 MB and 104 MB in the two runs. Real targets with out/, cache/ or node_modules/ will be much larger, and the registrations accumulate with every run.

clean leaves Git and engine state behind

  • ultrafuzz clean <run> --yes removes the run directory and refs/ultrafuzz/runs/<run>/source, and nothing else (packages/runtime/src/clean.ts).
  • Afterwards git worktree list still has 8 prunable entries, and the 8 ultrafuzz/<run>/<node> branches remain.
  • The project-root smithers.db still has the run in _smithers_runs (ultrafuzz-<run> | finished).
  • ultrafuzz run --run-id <run> then rebuilds the plan and the ~1 GB execution snapshot, and only at submission fails with:
    WORKFLOW_SUBMISSION_FAILED: workflow runner command failed (exit 1)
    DETACHED_ADMISSION_FAILED: Detached engine exited before admission (exit 4)
    code: RUN_EXISTS
    message: "Run already exists: ultrafuzz-<run>"
    
    The failed launch leaves a new partial run directory behind.
  • Not yet tested: whether the leftover branches and registrations also break git worktree add once the Smithers record is gone. Admission fails before any worktree is created.

Proposal

  • After verification publishes a task's artifacts, remove Ultrafuzz-owned paths (.ultrafuzz/, artifacts/) from the worktree. Alternatively, reap with Ultrafuzz's own cleanup instead of relying on Smithers' unsaved-work check. Keep a worktree only when it holds changes outside Ultrafuzz-owned paths.
  • clean should also remove the run's worktree registrations, ultrafuzz/<run>/* branches and Smithers run record.
  • Independently, run should reject a run ID that Smithers already knows before it builds the snapshot.

Acceptance

  • A successful run with keep_workspaces = false leaves no task worktrees and no stale git worktree list entries.
  • After clean <run>, run --run-id <run> launches successfully.

Related: #122 (artifact preservation before reaping, closed), #223 (worktree cleanup permissions, closed), #988 (clean leaking snapshots), #1148 / #1166 (source pinning).

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Real but moderatebugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions