Make a rare garbage-collector bug reproducible on demand (PERRY_GC_SCHEDULE_SEED) - #7317
Make a rare garbage-collector bug reproducible on demand (PERRY_GC_SCHEDULE_SEED)#7317jdalton wants to merge 1 commit into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis change replaces GC zeal with deterministic, rate-controlled seeded scheduling. It integrates scheduled safepoints with collection and evacuation policy, adds exit and signal diagnostics, provides tests and fuzzing tools, and updates GC reproduction documentation. ChangesSeeded GC schedule fuzzing
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Runtime
participant Safepoint
participant Schedule
participant Collector
participant Reporter
Runtime->>Schedule: resolve seed and rate
Safepoint->>Schedule: advance handled safepoint
Schedule-->>Safepoint: return collection selection
Safepoint->>Collector: perform scheduled moving minor
Collector->>Reporter: record forced collection
Reporter-->>Runtime: report counters on exit or failure
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
2467132 to
5d8ce73
Compare
There was a problem hiding this comment.
Actionable comments posted: 10
🧹 Nitpick comments (3)
crates/perry-runtime/src/gc/schedule.rs (1)
244-252: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAlign the "startup banner" claim with the actual announcement point.
The module documentation at Lines 331-334 describes layer 1 as "a startup banner, so the seed is in the log even if the failure mode is a hang or a
_exitthat runs no handler at all".resolved()runs the announcement lazily, at the first call site. For a mode-ON run, that is the first safepoint or the firstgc_force_evacuate_enabled()query. A hang or_exitbefore that point prints nothing, and no panic hook or signal handler is installed either.Consider resolving the configuration eagerly from GC initialization, or narrow the documentation claim to "the first safepoint" so an operator does not read a missing banner as "the seed was not set".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/perry-runtime/src/gc/schedule.rs` around lines 244 - 252, The startup-banner documentation does not match the lazy announcement in resolved(). Either eagerly resolve the configuration during GC initialization so publish_seed and announce run before early hangs or _exit paths, or narrow the layer-1 documentation to state that the banner appears at the first safepoint or gc_force_evacuate_enabled() query; preserve the existing seed publication behavior.crates/perry-runtime/src/gc/tests/schedule.rs (1)
276-283: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winRelease the GC root lock with a guard so a panic cannot leak the depth.
enter_gc_root_lock()andexit_gc_root_lock()are paired manually. Ifgc_safepoint_moving_minor()panics,exit_gc_root_lock()never runs, the root-lock depth stays non-zero for this thread, and every later collection on that thread is blocked. That converts one failure into a cascade of confusing failures in the same test binary.♻️ Proposed fix using a scope guard
let safepoints_before = gc_schedule_safepoints(); { let _schedule = ScheduleGuard::set(7, rate_threshold(1.0)); reset_thread_counter_for_test(); - super::super::roots::enter_gc_root_lock(); - gc_safepoint_moving_minor(); - super::super::roots::exit_gc_root_lock(); + struct RootLock; + impl RootLock { + fn enter() -> Self { + super::super::roots::enter_gc_root_lock(); + Self + } + } + impl Drop for RootLock { + fn drop(&mut self) { + super::super::roots::exit_gc_root_lock(); + } + } + let _lock = RootLock::enter(); + gc_safepoint_moving_minor(); }If the test support module already exposes a root-lock guard type, use it instead of the local shim.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/perry-runtime/src/gc/tests/schedule.rs` around lines 276 - 283, Update the test block around gc_safepoint_moving_minor to use the existing GC root-lock scope guard, if exposed by the test support module, instead of manually pairing enter_gc_root_lock and exit_gc_root_lock. Ensure the guard releases the lock during unwinding as well as normal completion, and remove the corresponding explicit exit call.docs/src/internals/gc-rooting-invariant.md (1)
271-279: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueKeep the
gc_schedule_fuzz.shargument syntax deterministic.The script accepts
<binary> [seed-count], butCLAUDE.mdstill says[seeds]. Update that line so the two docs use the actual positional argument semantics.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/src/internals/gc-rooting-invariant.md` around lines 271 - 279, Update the gc_schedule_fuzz.sh usage text in CLAUDE.md to describe the second positional argument as seed-count, matching the script’s actual <binary> [seed-count] semantics. Also review the usage reference in docs/src/internals/gc-rooting-invariant.md and changelog.d/7307-seeded-gc-schedule-fuzzing.md at the specified ranges; update any remaining [seeds] wording there to [seed-count], with no other changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CLAUDE.md`:
- Around line 145-146: Add a required CI workflow arm for the seeded GC schedule
OFF state, using a compiled program to test both an unset PERRY_GC_SCHEDULE_SEED
and PERRY_GC_SCHEDULE_RATE set without a seed. Verify both remain schedule-inert
while pressure-driven collections still occur, reusing the existing
scripts/gc_schedule_fuzz.sh or schedule test infrastructure where appropriate.
In `@crates/perry-runtime/src/gc/mod.rs`:
- Around line 778-784: The exit summary is emitted during per-thread teardown,
so SUMMARY_EMITTED can capture counts before other threads finish. Update the
report_exit_summary call in the exit path to emit only after all worker threads
have completed teardown—prefer the existing main-thread or final-thread
coordination mechanism—and preserve once-only reporting with complete safepoints
and scheduled_collections totals.
In `@crates/perry-runtime/src/gc/schedule.rs`:
- Around line 240-243: Add a required CI workflow arm that runs the GC schedule
tests or relevant test suite with PERRY_GC_SCHEDULE_SEED and
PERRY_GC_SCHEDULE_RATE unset, verifying their default/OFF behavior alongside
existing CI coverage. Anchor the change to the workflow job invoking the tests
and preserve the current configured-knob coverage.
- Around line 584-608: In the signal-handler teardown around the previous
handler lookup, restore SIG_DFL before entering the previous > 1 chaining path,
so the default disposition is installed before invoking the chained handler.
Keep the existing chained-handler call and early return, but remove the
later-only restoration structure so schedule_fault_handler cannot loop when the
chained handler returns.
- Around line 493-502: Update the previous-handler storage in
reinstall_signal_reporter_after to check old.sa_flags for libc::SA_SIGINFO
before saving old.sa_sigaction. Store 0 for handlers without SA_SIGINFO, while
preserving the existing self-chain prevention and storing the handler value only
when the flag is present.
In `@docs/src/internals/gc-rooting-invariant.md`:
- Around line 281-285: Update the paragraph beginning “A rate is not a
substitute for a schedule” to qualify the ~3/N confidence bound as applying only
to independent trials. State that repeated runs with a fixed seed or
deterministic schedule are correlated, so 0/N failures provide no statistical
bound, while preserving the guidance to vary collection timing.
In `@docs/src/internals/memory-model.md`:
- Around line 138-139: Update the PERRY_GC_SCHEDULE_SEED and
PERRY_GC_SCHEDULE_RATE documentation to describe the configured rate as
additional schedule density for minor collections only, applied when
gc_budgeted_due_trigger() reports no pressure-driven collection is due. Clarify
that pressure-driven collections still occur independently, so the rate is not
the total fraction of safepoints that collect, and replace the current “iff”
wording with this behavior.
In `@scripts/gc_instrument_smoke.sh`:
- Around line 119-129: Replace the `run_arm ... | tail -1` command substitutions
for `sched_retired`, `sched_repeat`, and `sched_other` with output capture that
does not use a pipeline, then explicitly check each `run_arm` exit status and
abort on failure before comparing results. Apply the same status-preserving
change to the other arms in this script that use the pipeline pattern, while
retaining extraction of the final output line.
- Around line 150-164: The strict schedule-density checks in the smoke fixture
can fail on low safepoint counts without demonstrating a broken rate knob.
Update the fixture to generate enough handled GC safepoints for distinct
retirement counts, or revise both failure paths around sched_retired,
nozeal_retired, and zeal_retired to report all three counts before exiting.
In `@scripts/gc_schedule_fuzz.sh`:
- Around line 53-59: Validate SEED_COUNT immediately after argument parsing as a
positive integer, rejecting zero and non-numeric values with an error and
nonzero exit. In the final summary around FAILED_SEEDS and the PASS output,
track executed runs via passed plus failed seeds and exit nonzero with a failure
message when that total is zero; only report PASS after at least one seed ran.
---
Nitpick comments:
In `@crates/perry-runtime/src/gc/schedule.rs`:
- Around line 244-252: The startup-banner documentation does not match the lazy
announcement in resolved(). Either eagerly resolve the configuration during GC
initialization so publish_seed and announce run before early hangs or _exit
paths, or narrow the layer-1 documentation to state that the banner appears at
the first safepoint or gc_force_evacuate_enabled() query; preserve the existing
seed publication behavior.
In `@crates/perry-runtime/src/gc/tests/schedule.rs`:
- Around line 276-283: Update the test block around gc_safepoint_moving_minor to
use the existing GC root-lock scope guard, if exposed by the test support
module, instead of manually pairing enter_gc_root_lock and exit_gc_root_lock.
Ensure the guard releases the lock during unwinding as well as normal
completion, and remove the corresponding explicit exit call.
In `@docs/src/internals/gc-rooting-invariant.md`:
- Around line 271-279: Update the gc_schedule_fuzz.sh usage text in CLAUDE.md to
describe the second positional argument as seed-count, matching the script’s
actual <binary> [seed-count] semantics. Also review the usage reference in
docs/src/internals/gc-rooting-invariant.md and
changelog.d/7307-seeded-gc-schedule-fuzzing.md at the specified ranges; update
any remaining [seeds] wording there to [seed-count], with no other changes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 77947a25-2bab-41b8-b287-31da05030b5f
📒 Files selected for processing (12)
CLAUDE.mdchangelog.d/7307-seeded-gc-schedule-fuzzing.mdcrates/perry-runtime/src/arena/quarantine.rscrates/perry-runtime/src/gc/mod.rscrates/perry-runtime/src/gc/policy.rscrates/perry-runtime/src/gc/schedule.rscrates/perry-runtime/src/gc/tests/mod.rscrates/perry-runtime/src/gc/tests/schedule.rsdocs/src/internals/gc-rooting-invariant.mddocs/src/internals/memory-model.mdscripts/gc_instrument_smoke.shscripts/gc_schedule_fuzz.sh
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CLAUDE.md`:
- Line 145: Update the PERRY_GC_SCHEDULE_SEED documentation to name Perry’s
process-exit teardown funnel as the source of seed reporting for _exit-based
exits, while describing atexit only as an additional reporting path. Preserve
the existing panic and signal-reporting paths and all other seed behavior.
- Around line 145-148: Condense the PERRY_GC_SCHEDULE_SEED and
PERRY_GC_SCHEDULE_RATE entries in CLAUDE.md to their concise runtime contract,
removing implementation rationale, reproduction guidance, and historical
context. Move that detailed narrative, including measurement guidance, to
changelog.d/7317-seeded-gc-schedule-fuzzing.md while preserving the documented
behavior and configuration semantics.
- Line 148: Update the documented invocation of scripts/gc_schedule_fuzz.sh to
use the optional argument name [seed-count] instead of [seeds], while preserving
the existing binary argument and surrounding guidance.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 91c7b636-1ffb-4ec0-ba19-591b2f2b1559
📒 Files selected for processing (12)
CLAUDE.mdchangelog.d/7317-seeded-gc-schedule-fuzzing.mdcrates/perry-runtime/src/arena/quarantine.rscrates/perry-runtime/src/gc/mod.rscrates/perry-runtime/src/gc/policy.rscrates/perry-runtime/src/gc/schedule.rscrates/perry-runtime/src/gc/tests/mod.rscrates/perry-runtime/src/gc/tests/schedule.rsdocs/src/internals/gc-rooting-invariant.mddocs/src/internals/memory-model.mdscripts/gc_instrument_smoke.shscripts/gc_schedule_fuzz.sh
🚧 Files skipped from review as they are similar to previous changes (10)
- docs/src/internals/gc-rooting-invariant.md
- crates/perry-runtime/src/arena/quarantine.rs
- crates/perry-runtime/src/gc/tests/mod.rs
- scripts/gc_instrument_smoke.sh
- crates/perry-runtime/src/gc/mod.rs
- docs/src/internals/memory-model.md
- crates/perry-runtime/src/gc/policy.rs
- scripts/gc_schedule_fuzz.sh
- crates/perry-runtime/src/gc/tests/schedule.rs
- crates/perry-runtime/src/gc/schedule.rs
|
The premise is right and it is the most useful framing anyone has put on this class:
That explains something we have been misreading. #7280's acceptance arms read 6, 8, 9 out of 30 across three runs of the same parent — we have been treating that as noise to work around, when it is really one schedule being sampled repeatedly. A seeded sweep is the right instrument, and it arrives at exactly the moment it is most needed: the owner has chosen to make statepoints the default and delete the shadow stack, and the soak deciding that is running now. Not merging yet, for two reasons:
What would make this land fast: the CI arm, and the two I have pointed the soak agent at this branch so it can use the sweep locally for schedule exploration without waiting on the merge — if it finds a failing seed on the statepoint arm, that is exactly the evidence the flip decision needs, and it would be a strong argument for landing this. |
5d8ce73 to
9c43d77
Compare
This comment was marked as outdated.
This comment was marked as outdated.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@changelog.d/7317-seeded-gc-schedule-fuzzing.md`:
- Around line 45-48: Update the 0/16 statistical statement in the changelog to
identify the confidence level and interval method used for the ~19% upper bound,
specifically describing it as a 95% Wilson upper bound.
In `@CLAUDE.md`:
- Around line 145-146: Update the CI workflow coverage for the GC scheduling
configuration to add required arms for an unset PERRY_GC_SCHEDULE_SEED and for
PERRY_GC_SCHEDULE_RATE configured without a seed. In each arm, verify
pressure-driven collections remain active while schedule-triggered collections
stay disabled, matching the documented OFF-state behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 3b65e0cf-c3cf-401f-8906-3df203dedc2b
📒 Files selected for processing (12)
CLAUDE.mdchangelog.d/7317-seeded-gc-schedule-fuzzing.mdcrates/perry-runtime/src/arena/quarantine.rscrates/perry-runtime/src/gc/mod.rscrates/perry-runtime/src/gc/policy.rscrates/perry-runtime/src/gc/schedule.rscrates/perry-runtime/src/gc/tests/mod.rscrates/perry-runtime/src/gc/tests/schedule.rsdocs/src/internals/gc-rooting-invariant.mddocs/src/internals/memory-model.mdscripts/gc_instrument_smoke.shscripts/gc_schedule_fuzz.sh
🚧 Files skipped from review as they are similar to previous changes (10)
- crates/perry-runtime/src/arena/quarantine.rs
- crates/perry-runtime/src/gc/policy.rs
- docs/src/internals/gc-rooting-invariant.md
- scripts/gc_schedule_fuzz.sh
- docs/src/internals/memory-model.md
- crates/perry-runtime/src/gc/mod.rs
- crates/perry-runtime/src/gc/tests/schedule.rs
- crates/perry-runtime/src/gc/schedule.rs
- crates/perry-runtime/src/gc/tests/mod.rs
- scripts/gc_instrument_smoke.sh
9c43d77 to
ca397ef
Compare
|
Follow-up in The signal-chain infinite loop was real, and my first pass was wrong to call it a false positive. Returning from a synchronous fault handler re-runs the faulting instruction, so if the chained quarantine handler also returned without resolving the fault, the disposition still pointed at us and we re-entered forever. The fix restores The The exit-summary gating I took as an improvement even though the original behavior was defensible. It now gates on a pure read, Where the OFF-state CI arm stands after thisThe required cargo-test coverage is complete as-is: |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/perry-runtime/src/native_handle.rs`:
- Around line 62-73: Update is_main_thread_or_unrecorded and its exit-summary
callers so an unrecorded MAIN_THREAD_ID does not allow a worker thread to claim
SUMMARY_EMITTED. Prefer ensuring js_native_handle_mark_main_thread() runs during
startup before workers begin, or make the teardown diagnostic skip callers while
MAIN_THREAD_ID remains 0; preserve acceptance only for the registered main
thread.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5fe7a9c1-d4d9-4981-bddc-14cde04fae59
📒 Files selected for processing (13)
CLAUDE.mdchangelog.d/7317-seeded-gc-schedule-fuzzing.mdcrates/perry-runtime/src/arena/quarantine.rscrates/perry-runtime/src/gc/mod.rscrates/perry-runtime/src/gc/policy.rscrates/perry-runtime/src/gc/schedule.rscrates/perry-runtime/src/gc/tests/mod.rscrates/perry-runtime/src/gc/tests/schedule.rscrates/perry-runtime/src/native_handle.rsdocs/src/internals/gc-rooting-invariant.mddocs/src/internals/memory-model.mdscripts/gc_instrument_smoke.shscripts/gc_schedule_fuzz.sh
🚧 Files skipped from review as they are similar to previous changes (10)
- docs/src/internals/gc-rooting-invariant.md
- crates/perry-runtime/src/gc/tests/mod.rs
- crates/perry-runtime/src/gc/policy.rs
- scripts/gc_instrument_smoke.sh
- docs/src/internals/memory-model.md
- crates/perry-runtime/src/arena/quarantine.rs
- crates/perry-runtime/src/gc/mod.rs
- crates/perry-runtime/src/gc/tests/schedule.rs
- scripts/gc_schedule_fuzz.sh
- crates/perry-runtime/src/gc/schedule.rs
|
@coderabbitai resolve |
There was a problem hiding this comment.
Actionable comments posted: 7
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
changelog.d/7317-seeded-gc-schedule-fuzzing.md (1)
142-144: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winReconcile the reported runtime test count before publishing this result.
This fragment reports
1670 passed, while the PR objectives report1687runtime library tests passing for the latest commit. Confirm whether the commands or feature sets differ. Update this fragment to the exact run that supports the current PR.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@changelog.d/7317-seeded-gc-schedule-fuzzing.md` around lines 142 - 144, Reconcile the runtime test-count claims in the changelog entry: verify the command and feature set used for the reported run against the latest PR result, then update the embedded test counts and run details to exactly match the current supporting execution. Remove or revise the stale comparison if it no longer reflects that run.docs/statepoint-gc-experiment.md (1)
933-942: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winUpdate the stale platform limitation.
The bullet says scanning is “macOS/Mach-O-only,” but the same document reports x86-64 and AArch64 Linux verification at Lines 589-613. State the current platform support, or label this bullet as an earlier prototype limitation.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/statepoint-gc-experiment.md` around lines 933 - 942, The bullet point stating "scanning is macOS/Mach-O-only" contradicts the platform verification reported elsewhere in the document (x86-64 and AArch64 Linux support at lines 589-613). Update this bullet to accurately reflect the current platform support including Linux platforms, or if this limitation applies only to an earlier prototype version, explicitly label it as such to clarify the scope and timeline of the constraint.
♻️ Duplicate comments (1)
docs/src/internals/gc-rooting-invariant.md (1)
281-285: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winQualify the
~3/Nbound.The paragraph says fixed-seed repetitions replay one schedule, but then applies the binomial
~3/Nbound to those repetitions. State that fixed-seed repetitions provide no statistical bound. Limit~3/Nto independent schedule or workload trials.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/src/internals/gc-rooting-invariant.md` around lines 281 - 285, The paragraph currently applies the ~3/N statistical bound to fixed-seed repetitions without clarifying that this bound is invalid for that scenario. Revise the text to explicitly state that re-running the same binary with fixed seeds provides no statistical bound on the true bug rate. Then limit the ~3/N bound statement to apply only when runs use independent schedules or workloads, making clear the distinction between replayed runs (which don't accumulate statistical evidence) and varied runs (which do).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/test.yml:
- Around line 1085-1091: Update the repository branch-protection required status
checks to include the existing GATING `gc-stress` context, ensuring failures
from the GC smoke-test workflow block merges. Do not alter the workflow test
logic.
In `@changelog.d/7219-registry-gc-unrooted-caches.md`:
- Around line 125-126: The `PERRY_GC_SCHEDULE_RATE` parameter is only effective
when `PERRY_GC_SCHEDULE_SEED` is set, so all documented test environments using
rate-1 scheduling must explicitly include the seed value. In
changelog.d/7219-registry-gc-unrooted-caches.md lines 125-126, add
`PERRY_GC_SCHEDULE_SEED=1` to both closure-call table rows. In
changelog.d/7227-regexp-receiver-rooting-and-alloc-re-audit.md lines 141-142,
add the seed to both regexp-receiver table rows. In
changelog.d/7253-gc-gate-main-line-run.md line 40, add the seed to the
14,373-minor verification environment. In
changelog.d/7270-rest-and-same-module-call-argument-rooting.md lines 55-56, add
the seed to both rest and same-module call rows. In
changelog.d/7280-optional-param-and-dynamic-construct-rooting.md line 67, add
the seed to the six-unit-reproducer environment. In
changelog.d/7317-seeded-gc-schedule-fuzzing.md lines 126-130, separate the
unseeded OFF-state trace from seeded rate-1 arms and explicitly name the seed
value for each ON-state trace environment.
In `@run_parity_tests.sh`:
- Around line 447-454: Update normalize_output’s sed filter to remove only the
known seeded GC schedule startup and exit diagnostic formats, rather than every
line beginning with “[gc-schedule]”. Preserve program output with that prefix
while continuing to strip the runtime-generated diagnostics before parity
comparison.
In `@scripts/gc_instrument_smoke.sh`:
- Line 3: Update the overview comment in gc_instrument_smoke.sh to include
PERRY_GC_SCHEDULE_RATE alongside PERRY_GC_SCHEDULE_SEED, so it documents both
scheduling controls exercised by the script.
- Line 214: The zero-probe diagnostic error message does not match the arm
number being tested. Locate the zero-probe error message that currently reports
"arm 4" and update it to report "arm 7" to align with the arm label shown in the
echo statement at line 214 for the quarantine test section, ensuring the failure
message correctly identifies which arm actually ran.
- Around line 252-253: Update the final summary near the retirement and
quarantine messages to avoid claiming zero retirements or program correctness
across all arms without corresponding assertions. Report the measured pressure
and rate retirement counts separately, and state the quarantine result
independently; alternatively, add explicit assertions for pressure_retired == 0
and Arm 7 probe-output correctness before making those claims.
In `@test-parity/gc_repsel_corpus.txt`:
- Around line 521-522: The measurement records in
test-parity/gc_repsel_corpus.txt document configuration requirements for
reproducing results but omit the PERRY_GC_SCHEDULE_SEED value that
PERRY_GC_SCHEDULE_RATE depends on. At lines 521-522 (the dynamic-construction
measurement record), add PERRY_GC_SCHEDULE_SEED=1 alongside POLLS=1 and RATE=1,
or explicitly note that the seed value is derived from the test harness. Apply
the same fix at lines 558-560 (the optional-parameter measurement record) to
ensure both documented scenarios include the complete seed specification needed
for reproducibility.
---
Outside diff comments:
In `@changelog.d/7317-seeded-gc-schedule-fuzzing.md`:
- Around line 142-144: Reconcile the runtime test-count claims in the changelog
entry: verify the command and feature set used for the reported run against the
latest PR result, then update the embedded test counts and run details to
exactly match the current supporting execution. Remove or revise the stale
comparison if it no longer reflects that run.
In `@docs/statepoint-gc-experiment.md`:
- Around line 933-942: The bullet point stating "scanning is macOS/Mach-O-only"
contradicts the platform verification reported elsewhere in the document (x86-64
and AArch64 Linux support at lines 589-613). Update this bullet to accurately
reflect the current platform support including Linux platforms, or if this
limitation applies only to an earlier prototype version, explicitly label it as
such to clarify the scope and timeline of the constraint.
---
Duplicate comments:
In `@docs/src/internals/gc-rooting-invariant.md`:
- Around line 281-285: The paragraph currently applies the ~3/N statistical
bound to fixed-seed repetitions without clarifying that this bound is invalid
for that scenario. Revise the text to explicitly state that re-running the same
binary with fixed seeds provides no statistical bound on the true bug rate. Then
limit the ~3/N bound statement to apply only when runs use independent schedules
or workloads, making clear the distinction between replayed runs (which don't
accumulate statistical evidence) and varied runs (which do).
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: f400a510-9796-4177-b65b-ed66f6000c5d
📒 Files selected for processing (30)
.github/workflows/test.ymlCLAUDE.mdchangelog.d/7196-gc-rooting-bug-instruments.mdchangelog.d/7219-registry-gc-unrooted-caches.mdchangelog.d/7227-regexp-receiver-rooting-and-alloc-re-audit.mdchangelog.d/7253-gc-gate-main-line-run.mdchangelog.d/7270-rest-and-same-module-call-argument-rooting.mdchangelog.d/7276-interned-string-cache-root-coverage.mdchangelog.d/7280-optional-param-and-dynamic-construct-rooting.mdchangelog.d/7311-dep-scale-corpus-and-root-reload.mdchangelog.d/7317-seeded-gc-schedule-fuzzing.mdcrates/perry-runtime/src/arena/quarantine.rscrates/perry-runtime/src/gc/mod.rscrates/perry-runtime/src/gc/policy.rscrates/perry-runtime/src/gc/schedule.rscrates/perry-runtime/src/gc/tests/fromspace_protect.rscrates/perry-runtime/src/gc/tests/schedule.rscrates/perry-runtime/src/gc/zeal.rscrates/perry-runtime/src/object/class_registry/construct.rsdocs/src/internals/gc-rooting-invariant.mddocs/src/internals/memory-model.mddocs/src/internals/rfc-rooting-by-construction.mddocs/statepoint-gc-experiment.mdrun_parity_tests.shscripts/gc_instrument_smoke.shtest-files/test_gap_gc_call_argument_rooting.tstest-files/test_gap_gc_regexp_receiver_rooting.tstest-files/test_gap_gc_rest_argument_rooting.tstest-files/test_gap_gc_same_module_call_argument_rooting.tstest-parity/gc_repsel_corpus.txt
💤 Files with no reviewable changes (1)
- crates/perry-runtime/src/gc/zeal.rs
🚧 Files skipped from review as they are similar to previous changes (5)
- crates/perry-runtime/src/gc/policy.rs
- crates/perry-runtime/src/gc/mod.rs
- crates/perry-runtime/src/gc/tests/schedule.rs
- crates/perry-runtime/src/arena/quarantine.rs
- crates/perry-runtime/src/gc/schedule.rs
✅ Action performedComments resolved. Approval is disabled; enable |
fd1f4e2 to
4bbf0b6
Compare
e16f010 to
a8df637
Compare
|
@coderabbitai resolve |
✅ Action performedComments resolved. Approval is disabled; enable |
|
@coderabbitai resolve |
✅ Action performedComments resolved. Approval is disabled; enable |
feae2dc to
4e9c364
Compare
de376aa to
628db9e
Compare
This comment was marked as outdated.
This comment was marked as outdated.
A PerryTS#7154-class bug is caught or missed by the GC schedule, not the bug, so re-running one binary re-runs one schedule and explores almost nothing. PERRY_GC_SCHEDULE_SEED=<u64> makes 'should this safepoint collect?' a deterministic pseudo-random function of the seed and a per-thread safepoint ordinal, at a density set by PERRY_GC_SCHEDULE_RATE (default 0.05), and a failing seed is a reproducer. scripts/gc_schedule_fuzz.sh sweeps seeds and prints a reproduce command per failure. Coexists with PERRY_GC_ZEAL (allocation-paced deterministic stress, PerryTS#7728): when both are set, zeal's pacing owns the loop-poll arm, the seed ticks only at safepoints zeal hands down, and a collection both would force is counted as zeal's, never twice. The poll arming word (PerryTS#7735) keeps its startup seed for either mode, so a schedule-only run cannot be silently disarmed. The forced-evacuation implication is unconditional for both modes, per PerryTS#7611. The seed is printed at startup, at exit (with the PerryTS#7604 liveness counters), on panic, and from a chaining async-signal-safe fatal-signal reporter that the from-space quarantine re-layers, so SEED + PROTECT_FROMSPACE reports both the seed and the precise fault site.
628db9e to
f3af2f0
Compare
|
Direction change since the previous comment, and a fresh push ( What forced the rethink: within an hour of the previous rebase, main landed #7729, which rebuilds zeal as allocation-paced ( What coexistence means concretely in this push
Validation: Known and deliberately not buried: CI triage against the previous head found one real defect owned by this PR — A follow-up PR will re-propose the zeal removal properly — porting the #7684 liveness counters to a neutral home, the #7604 verdict onto the schedule's endpoint, and #7728's allocation pacing into the schedule — stacked on this one so the diff collapses when this merges. |
This comment was marked as outdated.
This comment was marked as outdated.
PERRY_GC_SCHEDULE_SEED) — a failing schedule you can replay
Making a rare garbage-collector bug show up on demand
This adds a way to control when the garbage collector runs. A bug that normally shows up in about 1 run out of 60 can now be reproduced in about a second — and reproduced again the same way tomorrow, because you get a number back that replays the exact same run.
The problem
Perry has a garbage collector: it periodically finds memory the program is no longer using and reclaims it, sometimes moving still-live objects to new addresses while it does so. For that to be safe, the compiler has to tell the collector about every value the program is still holding. When it misses one, the collector either frees that value or moves it and leaves the program pointing at the old address. The program then keeps running with a bad pointer and crashes later somewhere unrelated, usually as
TypeError: value is not a function.Here is the awkward part. Whether that bug shows up at all depends entirely on whether a collection happens to land inside the small window where the value is unprotected. That is a property of the collector's timing, not of the bug. Normally collections happen tens of megabytes of allocation apart, so most runs sail straight past the window and finish fine.
That is why these bugs feel random, and why the usual response — run it again a bunch of times — does not actually work.
Why "just run it 100 times" tells you almost nothing
Say you have a fix and you want to confirm it. You run the program 120 times, see no failures, and call it fixed.
Statistically that is much weaker than it feels. With zero failures in
Nruns, the standard 95% confidence bound says the real failure rate could still be as high as roughly3/N. So 120 clean runs only prove the failure rate is below about 2.5% — and the bug we were chasing has a rate of about 1.7%. The evidence is entirely consistent with the bug still being there, untouched.To get real confidence by repetition alone you would need on the order of a thousand runs, every time you wanted to check anything.
The deeper issue is that repetition is not even exploring. Running one binary 60 times runs the same collection timing 60 times. You are not sampling 60 different scenarios; you are sampling one scenario 60 times.
What this adds
Set
PERRY_GC_SCHEDULE_SEEDto any number, and that number decides when collections happen. The decision is pseudo-random, so different seeds produce genuinely different collection timings and actually explore the problem space. But it is computed purely from the seed, so the same seed always produces the same timing — which means a seed that triggers a crash is the bug report. Anyone can replay it.A second variable,
PERRY_GC_SCHEDULE_RATE, controls how often collections happen, from "never" up to "at every opportunity". It defaults to 5%.There is also
scripts/gc_schedule_fuzz.sh, which tries a range of seeds and prints a ready-to-paste command for each one that fails.Does it actually work?
Yes. The test subject is Socket Firewall's
sfw-registry --help, which had been failing roughly 1 run in 60 — the kind of failure that had already cost days of investigation because nobody could summon it on demand.Same binary, same machine (macOS arm64), four runs at a time:
The failures fell into two consistent groups:
The first group is exactly the failure the investigation had been chasing. Seed 1 was then re-run five more times and failed all five, at the identical source location, in under a second each time:
So the thing that used to take an hour of re-running now takes a second, and comes with a copy-pasteable reproduction.
What it costs, and why half the sweep shows no result
Collecting more often is slower — roughly 5 to 10 times slower on this workload. That is why six of the twelve seeds above are listed as "cut off" rather than "passed": they had not finished within the 120-second limit, so we genuinely do not know whether they would have failed.
In practice this barely matters, because the seeds that find something fail in 1 to 2 seconds. The wall-clock time of a sweep is dominated entirely by the seeds that are not going to tell you anything. Turn the rate down or the timeout up depending on which you would rather buy.
How the timing decision actually works
Every so often a running Perry program reaches a point where it is safe to collect — the compiler guarantees that at these points it has told the collector about everything. Each thread counts these points: 1, 2, 3, and so on.
At each one, the program computes a hash of
(your seed, this counter value)and collects if the result falls below a threshold derived from the rate you asked for. That is the whole decision. It reads no clock, no memory address, and no thread identity, which is precisely what makes it replayable.A collection triggered this way also moves surviving objects rather than just sweeping. That matters: the entire class of bug being hunted is "the program kept a pointer to something that moved", so a mode that never moved anything would look busy and find nothing.
What it deliberately does not do
This only adds collections; it never removes or delays one that would have happened anyway. Turning it on can make a program collect more, never less.
It does not force a collection at moments the collector considers unsafe. If the program is in the middle of an allocation, inside a foreign-function call, or otherwise in a state where collecting would be wrong, the safepoint is skipped — and deliberately does not consume a counter tick, so the sequence stays aligned with points that could actually have collected.
It cannot create collection opportunities the compiler never emitted. If a program's hot loop has no safepoint in it, no seed can invent one, and the run will only collect at event-loop boundaries. The exit summary prints how many were reached, so you can check rather than assume.
Finally, a value that is not a valid number reads as off, not as seed 0. A typo must not silently give you a mode you did not ask for.
How reproducible this really is (the honest version)
For a single-threaded program, a seed replays exactly: same seed, same binary, same inputs, same collection timing, every time.
For a program using
perry/thread, each thread has its own counter, so each thread's timing is deterministic given that thread's own sequence of safepoints. What nobody can promise is that the operating system schedules those threads identically twice. So a multi-threaded reproduction is only as reproducible as its threading.The alternative — one shared counter across all threads — would be strictly worse, because then even a single thread's timing would depend on how the others interleaved. Per-thread is the version that keeps the guarantee meaningful where it can be kept.
You never lose the seed, even if the program dies badly
A fuzzer that finds a bug and loses the reproduction is worthless, so the seed is printed in four places: at startup, at exit, on a panic, and from a crash handler covering segfaults and aborts.
The crash handler is careful in two ways. It chains to whatever handler was already installed rather than replacing it, and the from-space memory-protection tooling re-installs it after setting up its own. That means when you combine this with
PERRY_GC_PROTECT_FROMSPACE=1— which is the natural pairing, since it turns a stale pointer into an immediate fault at the exact instruction — you get both the seed and the precise crash location, instead of one clobbering the other.The exit summary also reports how much the run actually did: how many safepoints were seen, how many collections were forced, how many objects moved. A clean run that did nothing looks very different from a clean run that did a lot, and now you can tell them apart.
Proving it is genuinely off by default
A debugging feature that quietly changes behaviour when it is switched off is worse than no feature. So this was measured rather than assumed.
With no seed set, the collector's diagnostic traces are byte-for-byte identical to the commit this branch is based on, across five different configurations on two test programs — including a 367-line trace in the normal configuration and a 6151-line trace with maximum collection pressure and memory protection both enabled.
Tests
gc/tests/schedule.rshas 12 tests covering both settings in both directions: that bad values are rejected, that the rate endpoints mean exactly "never" and "always", that the same seed picks the same 100,000 decisions twice running, that neighbouring seeds pick genuinely different ones (otherwise a sweep would be re-running one experiment under different names), that the measured collection rate matches the requested one at four different rates, and that a real safepoint collects, declines, or is skipped in the right circumstances.scripts/gc_instrument_smoke.shgains three end-to-end checks that gate the three claims this feature makes: that a middle rate really is in the middle (measured on the test program: 0 collections with no seed, 989 at 25%, 1230 at 100% — so it is a genuine range, not a second name for an endpoint), and that the same seed run twice produces exactly the same number.cargo test -p perry-runtimeon this branch: 1670 passed, 0 failed, run single-threaded, twice in a row.Two notes on scope
This no longer deletes anything. An earlier version of this pull request also removed the older stress-testing setting,
PERRY_GC_ZEAL. That turned out to deserve its own discussion, so it now has one in #7741. This branch adds the new capability alongside the existing one and changes nothing about how the collector behaves by default.The one real failure this branch owned is fixed separately. Running the conformance suite against this branch turned up a segmentation fault in
test_gap_buffer_own_props. It is fixed in #7747, which targetsmainrather than this branch, because the bug predates this work: reading a method off a Buffer without calling it produced a callable that remembered the method's name as a pointer into memory that had already been freed. Its sibling testtest_gap_buffer_own_prop_shadow_intrinsic_6405already crashes onmaintoday for the same reason.That is also a good illustration of why this pull request exists. Whether reading freed memory actually goes wrong depends on whether anything has reused it yet, so collecting more often here made an existing bug visible without this branch touching a single line of Buffer code. Once #7747 lands I will rebase this branch onto it and re-run the conformance suite.