test: stop process-global test state from failing unrelated tests - #208
Open
jmagar wants to merge 1 commit into
Open
test: stop process-global test state from failing unrelated tests#208jmagar wants to merge 1 commit into
jmagar wants to merge 1 commit into
Conversation
`cargo test` intermittently failed 7-9 tests in inventory, setup::doctor, and agent_observatory that had nothing to do with whatever was being changed. The failing set varied run to run, and CI never saw any of it, so the failures read as "your branch broke something" to whoever hit them. Three separate causes, all process-global state that `cargo nextest` hides by giving each test its own process. PATH override leaked. `container_probe_reports_unreachable_when_docker_ps_fails` (#205) replaced PATH with a bare tempdir and never restored it, so every later bare-name spawn in the binary resolved against a deleted directory and failed ENOENT. This reproduces at --test-threads=1, so it is a leak, not a race, and `#[serial]` alone would not have fixed it. Now prepends and restores via the file's own EnvGuard, and is `#[serial]` so it cannot collide with the three docker-stubbing tests in setup_tests.rs. PATH replacement had no scoped alternative. `collect_warns_when_optional_device_ commands_are_missing` legitimately wiped PATH to assert `ip`/`ss`/`df` are not installed — which took every concurrent test's spawns down with it. Adding `env::mask_test_programs`, which makes named programs resolve to an absent path so they spawn with the same NotFound an uninstalled binary gives. The test now says what it means, and no test in the tree replaces PATH any more. Projector tests asserted per-cycle values. `oversized_first_rows=1` and `"attempts":2` are overwritten by the projector's next cycle, so each was true for roughly one 10ms window and false forever after — no timeout could fix that, and `notify_projection_work` broadcasts on a process-global channel that any test's `insert_logs_batch` rings, forcing extra cycles. They now assert monotone facts: attempts >= 2, durable cursors, and that health reports the counter rather than what it currently reads. Deadlines also moved to named constants documenting the shared write lock, since every test pool's `init_pool` migrates while holding it. Verified on this tree: `cargo test --no-fail-fast` 2471/2471 (baseline 79adf1f: 8 failures) and `cargo nextest run` 3061/3061. Refs: syslog-mcp-g4frk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cargo testintermittently failed 7–9 tests ininventory,setup::doctor, andagent_observatory— modules unrelated to whatever was being changed — with the failing set varying run to run. CI never saw it, because CI runscargo nextest, which gives every test its own process. Three separate causes, all process-global state.Closes
syslog-mcp-g4frk.1. A
PATHoverride that leakedcontainer_probe_reports_unreachable_when_docker_ps_fails(added in #205) replacedPATHwith a bare tempdir and never restored it. Every later bare-name spawn in the binary then resolved against a since-deleted directory and failedENOENT.This reproduces at
--test-threads=1, so it is a leak, not a race —#[serial]alone would not have fixed it, since#[serial]only excludes other#[serial]tests and the override outlives the test either way.Now prepends to
PATHinstead of replacing it, restores through the file's ownEnvGuard, and is#[serial]so it cannot collide with the threedocker-stubbing tests insetup_tests.rs.2.
PATHreplacement had no scoped alternativecollect_warns_when_optional_device_commands_are_missinglegitimately wipedPATHto assertip/ss/dfare not installed — and in doing so stopped every concurrently running test from findingsh,git, and everything else.This is why "fall back to the real
PATHwhen the override misses" is not a viable fix: it breaks that test's intent. Instead this addsenv::mask_test_programs, which makes named programs resolve to a path that cannot exist, so they spawn with the sameNotFoundan uninstalled binary produces. The test now states what it means, its blast radius is four program names instead of every program, and no test in the tree replacesPATHany more.3. Projector tests asserted per-cycle values
oversized_first_rows=1and"attempts":2are both overwritten by the projector's next cycle, so each was true for roughly one 10 ms window and false forever after. No timeout could fix that — andnotify_projection_workbroadcasts on a process-global channel that everyinsert_logs_batchin the binary rings, so any test can drive any other test's projector through another cycle at any moment.They now assert monotone facts:
attempts >= 2, the durable cursors, and that health reports the counter rather than what it currently reads. The forward-progress property those tests exist for is still covered by the cursor assertion — withprojector_page_bytes = 1and a 16-byte row, a stalling page guard never advances the cursor.Deadlines also moved to named constants documenting why they are generous: every test pool's
init_poolruns the full migration set — VACUUM, CREATE INDEX, ANALYZE — while holding the one process-wide SQLite write lock.Verification
79adf1f4cargo test --no-fail-fastcargo nextest runSame machine, back to back.
cargo clippy --all-targetsandcargo fmt --checkclean; lefthook pre-commit gates pass.No production behaviour changes — every edit is test code or
#[cfg(any(test, feature = "test-support"))].