Stop git's background housekeeping from failing a revision - #42
Merged
Conversation
CI has been failing intermittently with
Errno::ENOENT: .../.git/objects/maintenance.lock
app/jobs/execute_instruction_job.rb relax_workspace_permissions
in unrelated tests — ExecuteInstructionJobTest one run, Templates::PickerTest
another, and passing in between on a docs-only commit. Not flakiness in the
tests: a real race in relax_workspace_permissions.
git 2.47 runs background maintenance automatically after commits, writing
transient lock files under .git. chmod_R lists a directory and then chmods each
entry it listed, so anything git removes in that window raises ENOENT and
aborts the entire walk. In CI that fails the build; in production it fails a
revision, because relax_workspace_permissions runs before every sandboxed roast
run on workspaces where git commands fire constantly. Local git 2.44 rarely
triggers it, which is why this has been invisible outside CI.
Fixed at both levels:
- Workspaces we create now set maintenance.auto=false and gc.auto=0 at git init
time. A tenant workspace is short-lived and small; it never needs to repack
itself, so the lock files should not exist in the first place.
- relax_workspace_permissions passes force: true, because workspaces created
before that config still exist in production and would otherwise keep racing.
Relaxation is best-effort by nature — a genuinely unchmoddable file surfaces
at the next operation with a clearer error than a half-finished walk.
Tests get the same treatment through GIT_CONFIG_COUNT in test_helper, which
injects the two settings into every git subprocess a test spawns without
touching the developer's global config. That part is not optional: the
Templates::PickerTest failure was inside Dir.mktmpdir's own cleanup, which no
application-side fix can reach.
The regression test models the exact window — chmod raising ENOENT for a path
the walk had already listed, which is what the kernel does when the file is
gone — and asserts the walk still relaxes the files that do exist. Verified to
error at relax_workspace_permissions without the force: change and pass with
it; an earlier version of this test passed either way, so it was rewritten
until it actually reproduced.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the intermittent CI failure that is currently blocking #41. Not a test flake — a real race that also affects production.
What's happening
git 2.47 runs background maintenance automatically after commits, writing transient lock files under
.git.chmod_Rlists a directory and then chmods each entry it listed — anything git removes in that window raisesENOENTand aborts the whole walk.In production that fails a revision.
relax_workspace_permissionsruns before every sandboxed roast run, on live workspaces where git commands fire constantly. Local git 2.44 rarely triggers it, which is why it has only ever shown up in CI.Evidence it's a race, not a bad test
ExecuteInstructionJobTestin one run,Templates::PickerTestin another.The fix, at both levels
maintenance.auto=falseandgc.auto=0atgit init. A tenant workspace is short-lived and small; it never needs to repack itself, so the lock files should not exist at all.relax_workspace_permissionspassesforce: true, because workspaces created before that config still exist in production and would otherwise keep racing. Relaxation is best-effort by nature: a genuinely unchmoddable file surfaces at the next operation with a clearer error than a half-finished walk.GIT_CONFIG_COUNTintest_helper, reaching every git subprocess a test spawns without touching your global config. Not optional: theTemplates::PickerTestfailure was insideDir.mktmpdir's own cleanup, which no application-side fix can reach.The regression test
Models the exact window —
chmodraisingENOENTfor a path the walk had already listed, which is what the kernel does when the file is gone — and asserts the walk still relaxes the files that do exist.Worth knowing: my first version of this test passed with the fix reverted, because
chmod_RcallsFile.chmod, notFileUtils.chmod, so the stub never fired. It was rewritten until it genuinely reproduced. It now errors atrelax_workspace_permissionswithout theforce:change and passes with it.Verification
562 runs / 0 failures, rubocop, brakeman clean.
Since the bug is intermittent, CI passing once here is suggestive, not proof. The root-cause half (no lock files created at all) is the part that should make it deterministic.