From 37b217fb444a8dab3b3bb0dfb156d64c8143de2b Mon Sep 17 00:00:00 2001 From: Largo Date: Mon, 10 Aug 2026 00:27:09 +0200 Subject: [PATCH 1/4] Keep the tests' Bundler runs out of OCRAN's own lockfile `bundle exec` exports BUNDLE_LOCKFILE with the absolute path of the lockfile it used, and the suite itself runs under `bundle exec rake test`, so every child it spawns inherits OCRAN's own Gemfile.lock by name. Bundler believes that variable over the Gemfile it is handed: bundle_exec_env, resolving a fixture Gemfile to find out what `bundle exec` sets, wrote the fixture's dependencies into OCRAN's lockfile instead of the fixture's own. Nothing in the test run noticed. What noticed was the next step in the same CI job that ran `bundle exec` - on Windows, the zlib portability build - which died in frozen mode over a lockfile that had `mylocal` in it and none of OCRAN's development gems. Linux and macOS run that step under plain `ruby` and so stayed green while carrying the same damaged lockfile. Clearing BUNDLE_LOCKFILE puts the lockfile back where Bundler would put it on its own, next to the Gemfile - for a fixture, the temporary directory it was copied into, which is removed with the rest of it. The lockfile is compared against its pre-run content after every such Bundler run, so the next variant of this names itself. Co-Authored-By: Claude Fable 5 --- test/test_ocra.rb | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/test/test_ocra.rb b/test/test_ocra.rb index 6d54a994..16e983e9 100644 --- a/test/test_ocra.rb +++ b/test/test_ocra.rb @@ -41,6 +41,24 @@ def exe_name(base) # Path to test fixtures. FixturePath = File.expand_path(File.join(File.dirname(__FILE__), 'fixtures')) + # OCRAN's own lockfile and its content before any test ran. The tests run + # Bundler against fixture Gemfiles, and a Bundler that has been pointed at + # the wrong lockfile rewrites this one instead of the fixture's. Nothing in + # the test run notices - the damage surfaces in whatever runs `bundle exec` + # afterwards, which in CI is a build step several minutes later. + OwnLockfile = File.join(OcranRoot, "Gemfile.lock") + OwnLockfileContent = File.exist?(OwnLockfile) ? File.binread(OwnLockfile) : nil + + # Asserts that running Bundler has not damaged OCRAN's own bundle. + def assert_own_lockfile_intact(what) + return if OwnLockfileContent.nil? + + assert OwnLockfileContent == File.binread(OwnLockfile), + "#{what} rewrote OCRAN's own lockfile (#{OwnLockfile}). " \ + "Bundler was run with a foreign Gemfile but the lockfile of " \ + "this bundle; restore it with `git checkout Gemfile.lock`." + end + # Create a pristine environment to test built executables. Files are # copied and the PATH environment is set to the minimal. Yields to # the block, then cleans up. @@ -134,10 +152,23 @@ def initialize(*args) # The environment `bundle exec` sets for the given Gemfile, or nil when # that bundle cannot be used here (Bundler missing, gems not installed), # so callers can skip rather than fail. + # + # BUNDLE_LOCKFILE is cleared rather than left alone. `bundle exec` exports + # the absolute path of the lockfile it used, and this suite itself runs + # under `bundle exec rake test`, so every child inherits OCRAN's own + # Gemfile.lock by name. Bundler believes that variable over the Gemfile it + # is handed, so resolving a fixture Gemfile here would write the fixture's + # dependencies into OCRAN's lockfile - leaving the tests green and every + # later `bundle exec` in the same job dead in frozen mode. Unset, Bundler + # puts the lockfile next to the Gemfile, which for a fixture means inside + # the temporary directory it was copied to. def bundle_exec_env(gemfile) dump = 'ENV.each { |name, value| puts "#{name}=#{value}" }' - output, status = capture_system({ "BUNDLE_GEMFILE" => gemfile.to_s }, + output, status = capture_system({ "BUNDLE_GEMFILE" => gemfile.to_s, + "BUNDLE_LOCKFILE" => nil }, "bundle", "exec", "ruby", "-e", dump) + assert_own_lockfile_intact("bundle exec against #{gemfile}") + return nil unless status&.success? env = output.lines.filter_map { |line| From 002efca242da509935d3db1f55e3dbf724b9a08f Mon Sep 17 00:00:00 2001 From: Largo Date: Mon, 10 Aug 2026 00:27:09 +0200 Subject: [PATCH 2/4] CI: fail the moment a test run dirties the checkout A test that lets Bundler write into the working tree leaves the suite green and breaks a later step with an error that names neither the test nor the fact that a test was responsible - the Windows jobs reported a frozen-mode Bundler failure in a build step minutes after the run that caused it. Checking the tree right after the tests puts the failure where the cause is. Tracked files only: bundle install populates vendor/bundle, and the builds leave stubs and packed executables behind. Co-Authored-By: Claude Fable 5 --- .github/check-clean-tree.sh | 23 +++++++++++++++++++++++ .github/workflows/test-on-push.yml | 12 ++++++++++++ 2 files changed, 35 insertions(+) create mode 100755 .github/check-clean-tree.sh diff --git a/.github/check-clean-tree.sh b/.github/check-clean-tree.sh new file mode 100755 index 00000000..7a682642 --- /dev/null +++ b/.github/check-clean-tree.sh @@ -0,0 +1,23 @@ +#!/bin/sh +# Fails when the test run modified a file under version control. +# +# The tests run Bundler against fixture Gemfiles. A Bundler that has been +# pointed at the wrong lockfile - `bundle exec` exports BUNDLE_LOCKFILE, and +# children inherit it - rewrites OCRAN's own Gemfile.lock with a fixture's +# dependencies. The test suite stays green; what dies is the next step that +# runs `bundle exec`, with a frozen-mode error that names neither the test +# that did it nor the fact that a test did it at all. +# +# Untracked files are ignored on purpose: `bundle install` populates +# vendor/bundle, and builds leave stubs and packed executables behind. +set -eu + +dirty=$(git status --porcelain --untracked-files=no) +if [ -n "$dirty" ]; then + echo "::error::the test run modified files under version control" + echo "$dirty" + git diff --stat + exit 1 +fi + +echo "working tree clean" diff --git a/.github/workflows/test-on-push.yml b/.github/workflows/test-on-push.yml index 4b9afe20..7f2b2672 100644 --- a/.github/workflows/test-on-push.yml +++ b/.github/workflows/test-on-push.yml @@ -32,6 +32,10 @@ jobs: bundle exec rake build bundle exec rake test + - name: Fail if the test run left the working tree dirty + shell: bash + run: bash .github/check-clean-tree.sh + - name: Build zlib fixture for cross-distro test env: OCRAN_DEBUG: "0" @@ -66,6 +70,10 @@ jobs: bundle exec rake build bundle exec rake test + - name: Fail if the test run left the working tree dirty + shell: bash + run: bash .github/check-clean-tree.sh + - name: Build zlib fixture for cross-system test run: ruby -Ilib exe/ocran test/fixtures/zlib/zlib.rb --output zlib_packed @@ -195,6 +203,10 @@ jobs: bundle exec rake build bundle exec rake test + - name: Fail if the test run left the working tree dirty + shell: bash + run: bash .github/check-clean-tree.sh + - name: Build zlib fixture for portability test run: bundle exec ruby -Ilib exe/ocran test/fixtures/zlib/zlib.rb --output zlib_packed.exe From c1399f978c87bb7ea6549b69e274f53e5ce6f319 Mon Sep 17 00:00:00 2001 From: Largo Date: Mon, 10 Aug 2026 00:28:21 +0200 Subject: [PATCH 3/4] CI: run the tests on branches that have a slash in their name `branches: ['*']` does not match `fix/...` or `feat/...`: a single star stops at a slash. Every branch this project uses was therefore merged untested, and master was the first place the tests ever ran on it - the last two merges both turned it red. workflow_dispatch as well, so a run can be asked for without a push. Co-Authored-By: Claude Fable 5 --- .github/workflows/test-on-push.yml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test-on-push.yml b/.github/workflows/test-on-push.yml index 7f2b2672..74aa7d0a 100644 --- a/.github/workflows/test-on-push.yml +++ b/.github/workflows/test-on-push.yml @@ -2,7 +2,12 @@ name: Run Tests on: push: branches: - - '*' + # '**' and not '*': a single star stops at a slash, so every branch + # named the way this project names them - fix/..., feat/... - was + # never tested until it landed on master, which is where the last two + # merges turned red. + - '**' + workflow_dispatch: jobs: run-tests-linux: strategy: From 5ba155031b985c0a1b46a64d4e9f9333478a89c6 Mon Sep 17 00:00:00 2001 From: Largo Date: Mon, 10 Aug 2026 07:44:37 +0200 Subject: [PATCH 4/4] CI: keep the checked-in bundle out of the clean-tree check vendor/ and .bundle/config are under version control here, and CI writes to both before a test runs: ruby/setup-ruby sets `deployment: true` in the config, and `bundle install` rewrites the vendored gems, whose shebangs and line endings differ per platform. On Windows that is some four thousand modified files, none of them anything a test did. Co-Authored-By: Claude Fable 5 --- .github/check-clean-tree.sh | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/.github/check-clean-tree.sh b/.github/check-clean-tree.sh index 7a682642..1410fcdc 100755 --- a/.github/check-clean-tree.sh +++ b/.github/check-clean-tree.sh @@ -8,15 +8,20 @@ # runs `bundle exec`, with a frozen-mode error that names neither the test # that did it nor the fact that a test did it at all. # -# Untracked files are ignored on purpose: `bundle install` populates -# vendor/bundle, and builds leave stubs and packed executables behind. +# Untracked files are ignored on purpose: the builds leave stubs and packed +# executables behind. So are vendor/ and .bundle/, which are checked in but +# describe the machine rather than the project: `bundle install` rewrites the +# vendored bundle (shebangs and line endings differ per platform) and +# ruby/setup-ruby writes `deployment: true` into the config, both before +# anything of ours has run. set -eu -dirty=$(git status --porcelain --untracked-files=no) +dirty=$(git status --porcelain --untracked-files=no \ + -- . ':(exclude)vendor' ':(exclude).bundle') if [ -n "$dirty" ]; then echo "::error::the test run modified files under version control" echo "$dirty" - git diff --stat + git diff --stat -- . ':(exclude)vendor' ':(exclude).bundle' exit 1 fi