Bug: macOS test process exits with SIGTRAP after passing assertions

[lamemustafa]

The macOS native job for #247 at 6e899260b52b5df30ee9db70dae8166766d7de34 failed after one test printed a passing libtest summary and its process then exited with SIGTRAP (signal 5). The suite correctly failed; no test was skipped or retried.

Evidence: run 34229575462, native macOS job. The affected test is db::tally_mirror::tests::reviewed_setup_consumption_authority_is_immutable. At 13:08:44 UTC on 8 September 2026, its captured stdout reports one passed test, then nextest reports process signal 5. Final suite: 1,117 run, 1,116 passed, one failed, zero skipped; step exit 100. The log contains no crash stack.

Root cause is unverified. #247 does not change this DB test. #250 introduced process-per-test execution with nextest; the relationship to this failure is unproven. Final #250 native checks passed on both platforms. A bounded local comparison on the retained pre-#250 bd8d594 binary ran this exact test directly in 100 separate processes, without nextest: all printed one passed test and exited zero. A second retained local binary built at 5b561313, application-source-equivalent to #247 6e899260, also passed 100 direct isolated-process repetitions with exit zero. Its SHA-256 stayed 6189d92080c39e2950fc8ecb6f3399cf36a44abcd5fa2dc5a00ab738a67d879d; no recompilation occurred. Neither binary was the hosted binary. These 200 successful local repetitions do not reproduce or explain the hosted crash. The complete affected DB module and Cargo.lock have identical Git blobs across the older source, merged #250 and affected #247 source.

Acceptance:

• Capture a macOS crash report/backtrace or reproduce the nonzero process exit on the affected source/runtime; distinguish assertion completion from successful process exit.
• Compare direct libtest process execution and nextest with the same binary and source where feasible.
• Identify the causal boundary before changing teardown or runner policy, then add the smallest regression check and a Rectify PR if a code fix is needed.
• Preserve every test, explicit doctests, no-retry policy and required native platform checks. A rerun passing by itself is not a fix.

This issue tracks diagnosis in the CI lane. It does not authorize changes to #247's product workflow or treat the failure as harmless.

Second hosted attempt on the unchanged source also failed: job 102081104766. This time sync::snapshot::tests::stale_commit_pending_checkpoint_terminalizes_and_closes_staging_batch printed one passed test, then exited with SIGTRAP5 at 13:32:37 UTC. Suite: 1,117 run, 1,116 passed, one failed, zero skipped; exit100. A different affected test makes a shared process-exit path worth investigating, but no causal claim is established.

Diagnostic PR #254 adds minimized fresh macOS crash-stack capture on top of the exact affected application source. Instrumented run 34233989377 executes the full existing CI workflow at 9b7dcd876eb7c9b9011dedb207de2fddedac82dd. The collector passed focused privacy/parser tests and captured a real disposable local SIGTRAP stack. Raw reports, register contents and process paths are excluded from the artifact. No product tests, retries, serialization, timeout or required coverage changed. This run is for diagnosis, not a blind retry or a fix claim.

Bounded diagnosis update: run 34233989377 passed all full CI jobs with no fresh crash reports. Run 34235523547 also passed full CI and 250 executions of each affected test (500 total), without retries and with identical before/after binary SHA c0f71a9ef4ca9b052e7538f871bf9a87171283ddd14d99fe34b9ea5e7e65933b; no crash report was produced. These are non-reproductions, not fixes.

Hosted control run 34237062493 successfully captured a deliberately crashed disposable process with SIGTRAP and symbolicated frames. Thus the collector is verified on a hosted Mac. The control stack is not the application-failure stack. Both original failures and both instrumented full runs used macos-26-arm64 image20260831.0337.3. No retained crash stack exists among the original run's artifacts. #253 remains open and diagnostic #254 remains separate; no SQLx/runtime changes or extra blind cycles are justified by the current evidence. A new #247 application head requires a fresh diagnostic rebase/audit before reusing this candidate's claims.

[lamemustafa]

Additional observed failure on master 3851f00b9b51fcea7ce4785a970ad5944bed184a: run34243807340, macOS native test db::tally_mirror::tests::explicit_window_abandon_clamps_clock_rollback_and_is_terminal_immutable printed one passing libtest result and then exited with SIGTRAP5 at15:21:27UTC. Full suite:1109run,1108passed,1failed,0skipped; nextestexit100.

Both packaging jobs and Windows native passed. This supplies a third affected test with the same observed post-assertion exit pattern; it does not establish the cause or attribute it to packaging. No blind rerun was dispatched. The source did not include the separately reviewed crash collector, so this run does not provide its minimized IPS artifact.

[lamemustafa]

Ownership update: backlog orchestration owns evidence-led diagnosis and any resulting fix; the CI performance task retains isolated performance experiments and provides diagnostic integration review. Product PR owners retain their branches.

#261 is now merged as 37c8a9bc2585e2b5355b6e82891bab7d6d2decc4; #254 is closed as superseded, preserving its historical evidence. #261 provides fresh minimized macOS crash stacks and a hosted disposable control, not an application-crash fix.

The next diagnostic gap is exact native test-binary retention with source/run/attempt and image UUID/architecture linkage. Existing package artifacts contain release executables, not the failing nextest executable. A bounded failure-only diagnostic companion is being scoped before the product owner performs a purposeful integration run. Debug information is disabled in the current native profile; retained binaries alone will not supply missing line tables. Root cause remains unverified; no pool teardown change, blind stress loop, lucky-green retry, or weakened gate is authorized as a substitute for causal evidence.

[lamemustafa]

Diagnostic companion #265 merged as 025d7fe689b049073381f15301593abb98749d26, after clean exact-head review, normal CI34271940974 on both native/package platforms, and disposable control34271944958. It now retains only exact image-UUID/architecture-linked native executables with digest and checkout/run provenance after a native failure. The control artifact10074181237 was independently byte-verified; a different image is refused.

This issue remains open; the root cause is unverified. Backlog task remains investigation owner, with performance task providing infrastructure review and historical evidence. Next actionable evidence is a fresh naturally occurring native SIGTRAP with its minimized report and matching executable from the same instrumented job. Preserve both artifacts and inspect the faulting thread/image offsets before proposing a fix. No speculative pool-close change, blind stress run, or lucky-green retry is justified by the current record.

[lamemustafa]

Additional occurrence tracked here; no duplicate issue is needed.

Post-merge master ac50b986815e8f1cd94d8b814a337fbcb1ce3c3d has a failed macOS native job 102407936751. The CI owner retained a SIGTRAP report from the existing db::tally_mirror::tests::commit_rejects_open_attempt_until_proof_bound_cleanup_completes test: 1,139 of 1,140 passed, one failed, zero skipped.

The macOS package job passed, while Windows jobs were still running at this status check. Package/cache-seed success does not qualify the complete post-merge run. This occurrence does not establish a cause or attribute the failure to #271. No rerun or runtime workaround has been requested.

#253 remains open under the existing backlog investigation ownership. Further crash investigation is paused; this update only reconciles the CI failure with the existing issue. The performance task retains cache qualification and its separate draft #272 experiment.

[lamemustafa]

A bounded native repair is now merged in #315 at b4c029d6f525badcca466ea9ee4a54e52e22677e (reviewed head b2772835896ba25b9386339479a295e1e0e053c5).

The later #315 failing artifact contained the real post-assertion teardown stack mfm_free → init_thread_stop → init_thread_destructor → pthread cleanup, while another thread was in SQLCipher exit cleanup. The retained Mach-O contained the corresponding static OpenSSL functions; the older #272 artifact showed the same teardown family. The pinned libsqlite3-sys build script confirmed that the old macOS Cargo feature actually selected vendored OpenSSL.

The merged change selects the default CommonCrypto provider on macOS, preserving the existing vendored provider on other targets. The default rebuilt macOS artifact has no matching OpenSSL teardown symbols. Encrypted synthetic database reads passed old→new and new→old, fresh wrong-key rejection passed, and the named test passed in 20 fresh local processes. The final head passed full required CI including native macOS/Windows and both bundle checks. No retries, skipped tests, runner policy or dependency versions were changed.

This removes the concrete teardown path observed in the later failure and supports the provider boundary as a strong causal candidate. It does not establish that every earlier SIGTRAP shared that cause, and a green rerun alone is not being used as closure evidence. Keeping this diagnostic issue open for that remaining historical attribution; the original reports and no-retry requirements remain intact.

[lamemustafa]

Measured rather than reasoned about, because this has sat with an unverified root cause since September. Three findings, none of which closes it, but together they change what the next step is.

1. The diagnostic postdates the only occurrence by one day. scripts/collect-macos-test-crashes.py landed 2026-09-09 in #261. This failure was 2026-09-08. So "the log contains no crash stack" is a fact about a run that happened before anything was collecting stacks — not evidence that the crash leaves none.

The collector now runs under if: always(), so it fires whether the suite passes or fails, waits 30s when tests failed, and uploads native-crash-stacks-macos with if-no-files-found: error. If this recurs, there will be a stack.

2. It has not recurred in 100 CI runs since. Of those, 81 succeeded, 9 were cancelled and 9 failed. I opened every failed run's macOS native job: zero mentions of SIGTRAP, signal 5, SIGSEGV or abort. Every macOS failure is accounted for by something else — most are tonight's surface_file_changed reseal mismatches (which fail "Tally portable core" in the same run, a recognisable signature), and one is a genuinely failing test, a_party_difference_echoes_the_source_spelling_not_its_catalog_binding, since fixed in #294.

3. Not reproducible locally in isolation. I ran db::tally_mirror::tests::reviewed_setup_consumption_authority_is_immutable 60 times on macOS with the pinned 1.96.0 toolchain: 60 clean exits, zero non-zero.

That last one is weak evidence and I want to be clear why: CI runs it under cargo nextest run --test-threads num-cpus with process-per-test, and I ran it under plain cargo test in isolation. So I did not reproduce the execution model, only the test. Anyone trying again should install cargo-nextest@0.9.143 (the pinned version) and run the full workspace under load — the original failure came 866 tests into a 1,495-test run, which is exactly where resource pressure lives.

Suggested disposition. Not closeable — a one-off crash with no stack and no reproduction is not the same as a fixed one. But the useful state is different from "unverified": the crash is now armed for diagnosis, and the thing to do is nothing until it recurs, then read native-crash-stacks-macos from that run rather than re-deriving any of this.

Worth labelling to that effect so the next person does not repeat the 100-run sweep. If it has not recurred by a fixed date, closing as not-reproducible with this comment as the record would be honest — the evidence for "no longer happening" is 100 runs, which is not proof but is more than it had.

[lamemustafa]

Recurrence sweep since the last update (2026-09-15T23:12Z), measured on 2026-09-17 about 03:50 IST.

• 127 CI workflow runs were created in that window. Their Native checks (macos-latest) jobs concluded: 68 success, 3 failure, 11 cancelled, 2 not finished. The remaining runs have no macOS native job.
• A successful macOS native job runs cargo nextest run --workspace --retries 0. A post-assertion SIGTRAP fails that step, so each of the 68 is a full-suite run without one. Sampled job 104994743175: "1570 tests run: 1570 passed".
• The 3 failed macOS jobs (104993185146, 104823313372, 104817698014) have no nextest summary line; each log shows an install-action error line, so they don't count as evidence either way. I grepped all 13 failed jobs in the 4 failed runs for SIGTRAP|signal 5|SIGSEGV|SIGABRT: zero matches. Each log was fetched, with over 2,000 lines per macOS job.

With the earlier 100-run sweep and the #315 CommonCrypto change, the post-assertion SIGTRAP has not been seen since. This sweep does not establish that the earlier occurrences shared #315's cause. The crash collector (native-crash-stacks-macos) still runs on every macOS job, so any recurrence will come with a stack.

Suggested disposition, for the owner to decide: close as not recurring after the #315 provider change, pointing to this comment and the previous one, and reopen with the retained stack if it recurs.

[lamemustafa]

Third recurrence sweep, covering everything after the previous one (2026-09-16T22:21Z). Measured 2026-09-21 about 15:30 IST.

• 204 ci.yml runs were created in the window. For one run the jobs listing call failed, so the counts below are at least these numbers.
• Native checks (macos-latest): 136 success, 5 failure, 14 cancelled, 1 not finished. A successful job runs the full workspace under cargo nextest run --retries 0, so each success is a full-suite run with no post-assertion SIGTRAP.
• I fetched the complete log of each of the 5 failed jobs (~2,800 lines each) and grepped for SIGTRAP|signal 5|SIGSEGV|SIGABRT: 0 matches. All 5 are Dependabot PRs (Bump rust_xlsxwriter from 0.99.0 to 0.99.1 in /src-tauri #542–Bump the frontend-development group with 3 updates #547) that failed on reseal/notice drift, not on a test crash.

Cumulative since #315 (CommonCrypto provider on macOS): no recurrence across three sweeps. This still does not prove the earlier occurrences shared #315's cause. The crash collector still runs on every macOS job, so a recurrence would come with a stack.

Next step, for the owner: close as not recurring after #315, citing this and the two previous sweeps, and reopen with the retained native-crash-stacks-macos artifact if it recurs. I have not closed it, because the previous comment left that decision to the owner.

[lamemustafa]

Closing by owner decision: no recurrence since #315 (merged 2026-09-12) across three recurrence sweeps recorded in this thread, the latest on 2026-09-21. Reopen if a macOS native test process exits with SIGTRAP after passing assertions again.