Skip to content

Stop git's background housekeeping from failing a revision - #42

Merged
pstrzalk merged 1 commit into
mainfrom
fix/git-maintenance-chmod-race
Aug 27, 2026
Merged

Stop git's background housekeeping from failing a revision#42
pstrzalk merged 1 commit into
mainfrom
fix/git-maintenance-chmod-race

Conversation

@pstrzalk

Copy link
Copy Markdown
Owner

Fixes the intermittent CI failure that is currently blocking #41. Not a test flake — a real race that also affects production.

What's happening

Errno::ENOENT: .../.git/objects/maintenance.lock
  app/jobs/execute_instruction_job.rb relax_workspace_permissions

git 2.47 runs background maintenance automatically after commits, writing transient lock files under .git. chmod_R lists a directory and then chmods each entry it listed — anything git removes in that window raises ENOENT and aborts the whole walk.

In production that fails a revision. relax_workspace_permissions runs before every sandboxed roast run, on live workspaces where git commands fire constantly. Local git 2.44 rarely triggers it, which is why it has only ever shown up in CI.

Evidence it's a race, not a bad test

  • It lands in different, unrelated testsExecuteInstructionJobTest in one run, Templates::PickerTest in another.
  • A docs-only commit flipped CI from green to red six minutes apart.
  • The failing code was last touched 2026-06-12 and is untouched by Upgrade to RubyLLM 2.0 #41.

The fix, at both levels

  • Root cause — workspaces we create set maintenance.auto=false and gc.auto=0 at git init. A tenant workspace is short-lived and small; it never needs to repack itself, so the lock files should not exist at all.
  • Resiliencerelax_workspace_permissions passes force: true, because workspaces created before that config still exist in production and would otherwise keep racing. Relaxation is best-effort by nature: a genuinely unchmoddable file surfaces at the next operation with a clearer error than a half-finished walk.
  • Tests — same two settings injected via GIT_CONFIG_COUNT in test_helper, reaching every git subprocess a test spawns without touching your global config. Not optional: the Templates::PickerTest failure was inside Dir.mktmpdir's own cleanup, which no application-side fix can reach.

The regression test

Models the exact window — chmod raising ENOENT for a path the walk had already listed, which is what the kernel does when the file is gone — and asserts the walk still relaxes the files that do exist.

Worth knowing: my first version of this test passed with the fix reverted, because chmod_R calls File.chmod, not FileUtils.chmod, so the stub never fired. It was rewritten until it genuinely reproduced. It now errors at relax_workspace_permissions without the force: change and passes with it.

Verification

562 runs / 0 failures, rubocop, brakeman clean.

Since the bug is intermittent, CI passing once here is suggestive, not proof. The root-cause half (no lock files created at all) is the part that should make it deterministic.

CI has been failing intermittently with

  Errno::ENOENT: .../.git/objects/maintenance.lock
    app/jobs/execute_instruction_job.rb relax_workspace_permissions

in unrelated tests — ExecuteInstructionJobTest one run, Templates::PickerTest
another, and passing in between on a docs-only commit. Not flakiness in the
tests: a real race in relax_workspace_permissions.

git 2.47 runs background maintenance automatically after commits, writing
transient lock files under .git. chmod_R lists a directory and then chmods each
entry it listed, so anything git removes in that window raises ENOENT and
aborts the entire walk. In CI that fails the build; in production it fails a
revision, because relax_workspace_permissions runs before every sandboxed roast
run on workspaces where git commands fire constantly. Local git 2.44 rarely
triggers it, which is why this has been invisible outside CI.

Fixed at both levels:

- Workspaces we create now set maintenance.auto=false and gc.auto=0 at git init
  time. A tenant workspace is short-lived and small; it never needs to repack
  itself, so the lock files should not exist in the first place.
- relax_workspace_permissions passes force: true, because workspaces created
  before that config still exist in production and would otherwise keep racing.
  Relaxation is best-effort by nature — a genuinely unchmoddable file surfaces
  at the next operation with a clearer error than a half-finished walk.

Tests get the same treatment through GIT_CONFIG_COUNT in test_helper, which
injects the two settings into every git subprocess a test spawns without
touching the developer's global config. That part is not optional: the
Templates::PickerTest failure was inside Dir.mktmpdir's own cleanup, which no
application-side fix can reach.

The regression test models the exact window — chmod raising ENOENT for a path
the walk had already listed, which is what the kernel does when the file is
gone — and asserts the walk still relaxes the files that do exist. Verified to
error at relax_workspace_permissions without the force: change and pass with
it; an earlier version of this test passed either way, so it was rewritten
until it actually reproduced.
@pstrzalk
pstrzalk merged commit 7f4b058 into main Aug 27, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants