perf(gc): whole-block in-place promotion of a fully-live young generation (#7742)

[proggeramlug]

Closes part of #7742.

When a copying minor's nursery is (near-)entirely live, relabel its blocks as
old-gen instead of evacuating them object by object.

bench	main	this PR	node
retain	0.81 s / 326 MB	0.53 s / 316 MB	0.12
retain_wide	1.33 s / 447 MB	1.07 s / 446 MB	0.15
retain1	0.39 s / 148 MB	0.29 s / 133 MB	—
retain_wide1	0.38 s / 174 MB	0.27 s / 155 MB	—
deeplist	0.30 s / 117 MB	0.24 s / 97 MB	0.09
churn / churn_alloc / push_cls	0.41 / 0.37 / 0.35	0.41 / 0.37 / 0.35
push_num / churn_read / cycles	0.13 / 0.02 / 0.19	0.14 / 0.02 / 0.19
tree / tree_wide	1.64 / 2.10	1.63 / 2.10

Best-of-5 wall clock on the pinned quiet M1 mini (load ~1.6), interleaved,
stdout byte-verified against node --experimental-strip-types before timing.
Promotion cost per object 243 ns → 101 ns, derived from the GC trace's own
promoted-object count and pause.

Why the old path was pure overhead here

retain.ts builds a 3M-element array of records and drops none. Its trace
measures a young-survival ratio of 1.000 on every copying minor, and its
four promoting minors moved 2,097,155 objects in 509.7 ms. Each move paid for a
fresh arena_alloc_gc_old, a memcpy, a layout_transfer,
old_page_account_promoted_object, the move hooks, a forwarding stub, and the
rewrite of every referring slot — to put an object somewhere it had no reason
to be.

Design notes worth reviewing

• The decision is a measurement. Block liveness is not knowable before the
  trace and Eden blocks are recycled at offset 0, so it is per cycle from the
  previous cycle's ratio. What makes that sound is that a promoting cycle
  still traces, so it re-measures — a workload that flips from live to
  garbage pays one nursery of retained garbage and the policy turns itself off.
  A running 32 MB cap bounds the steady state the per-cycle correction misses.
  The measured population is bimodal by three orders of magnitude (1.000 on
  retain/deeplist, ≤ 0.004 on churn/push/cycles).
• Old ⟹ TENURED (perf(gc): write barriers cost 16% on an all-numeric store workload — elide on provably-non-pointer stores #7511) is upheld by stamping every header on a promoted
  block, dead ones included, and asserted end to end.
• Three remembered-set passes are skipped on a promoting cycle, and it is a
  proof rather than a heuristic: after the retag no address classifies as
  Nursery (every in-use young block was taken), so with an empty malloc
  registry there is no child any of them could remember.
  debug_assert_no_remembering_possible re-derives it from the heap. Most of
  the win is here.
• Pacing is the part that made the first working version a 0.81 → 1.52 s
  regression: promotion hands Eden's blocks away, so the arena-bytes trigger's
  runway collapses. The capacity is returned as one-shot trigger headroom
  rather than as re-reserved blocks (the eager form fixes pacing too but maps
  memory the program may never reach — retain_wide peak RSS 470 vs 447 MB).
• classify_arena resolved the page-generation range twice per visited
  slot; it now resolves once and answers the header's classification from the
  range base, which is also the guard that keeps a garbage candidate at a range
  start from reading the unmapped page below.

Validation

• cargo test -p perry-runtime --release: 1982 passed, 0 failed.
• gc-handoff/apps/iso_miss.ts → checksum 437840 misses 0, plain and under
  PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=800 and
  PERRY_GC_VERIFY_EVACUATION=1. It is an interpreter, so it takes the
  ordinary copying path on all 50 of its minors and still retires 50 quarantine
  sets — the instrument's coverage is unchanged, and it does not cover the new
  path.
• What does cover the new path: PERRY_GC_FROMSPACE_SCAN_ABORT=1 ./n_retain
  over 4 promoting cycles (up to 2,111,418 objects / 21,085,616 words per
  cycle) reports missing_rewrites=0 dangling=0 never_dirty=0 lost_dirty=0 dirty_but_missed=0 on every cycle — the last three are exactly the "the
  remembered set is missing an edge" counters the skipped passes could break.
• gc_ratchet (7 repeats, shared_ci): every gated semantic cell is
  identical to origin/main measured on the same host, 12_large_live_set
  included. The six cells it reports as REGRESSION are reported identically by
  origin/main itself — pre-existing drift against the pinned artifact. The
  corollary is that no ratchet probe runs a promoting cycle, so the ratchet does
  not yet cover this path.
• check_file_size.sh, addr_class_inventory.py, cargo fmt --all -- --check
  all clean.

Not closed

retain is still 4.4× Node and retain_wide 7.1×, so #7742 stays open. The
residue is no longer promotion bookkeeping: on a retain_live_big profile the
remaining GC time is the remembered-set scan over the array's dirty element
slots (26%) and one full collection that reclaims nothing (28%).

Summary by CodeRabbit

• New Features

  • Added whole-block in-place promotion for high-survival young-generation memory while preserving object addresses.
  • Uses measured survival rates and configurable controls to select promotion.
  • Added promotion metrics, diagnostics, and capacity accounting.

• Performance

  • Reduces unnecessary remembered-set processing and object relocation when safe.
  • Retains copying collection for lower-survival workloads.

• Tests

  • Added coverage for thresholds, configuration, accounting, metrics, and address preservation.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e6381aac-951f-4f27-9d99-0f2d9588c8b8

📥 Commits

Reviewing files that changed from the base of the PR and between 49c6f58 and 7d89800.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (4)

• CLAUDE.md
• Cargo.toml
• crates/perry-runtime/src/gc/policy.rs
• crates/perry-runtime/src/gc/telemetry.rs

🚧 Files skipped from review as they are similar to previous changes (2)

• crates/perry-runtime/src/gc/telemetry.rs
• crates/perry-runtime/src/gc/policy.rs

---

📝 Walkthrough

Walkthrough

The runtime adds measured whole-block in-place promotion for highly live young-generation blocks. It retags blocks, indexes live objects, integrates promoted objects into copying collection, updates pacing and telemetry, and adds policy and end-to-end tests.

Changes

Whole-block in-place promotion

Layer / File(s)	Summary
Promotion policy and accounting crates/perry-runtime/src/gc/promote_in_place.rs, crates/perry-runtime/src/gc/mod.rs, crates/perry-runtime/src/gc/policy.rs, changelog.d/7744-gc-whole-block-in-place-promotion.md	Promotion uses measured young-survival ratios, configuration controls, retained-garbage limits, counters, and capacity credits. Full collections reset retained-garbage accounting.
Arena retagging and finalization crates/perry-runtime/src/arena/*, crates/perry-runtime/src/gc/layout.rs, scripts/addr_class_allowlist.txt	Young blocks receive temporary PromotedYoung metadata. Finalization stamps headers, scans objects, registers live page runs, installs blocks in old generation, and resets young regions.
Copying collector integration and diagnostics crates/perry-runtime/src/gc/copying.rs, crates/perry-runtime/src/gc/telemetry.rs	The collector classifies and marks promoted objects without relocation, conditionally skips remembered-set work, records survival and promotion statistics, adjusts accounting, and serializes new diagnostics.
Policy and end-to-end tests crates/perry-runtime/src/gc/tests/*, crates/perry-runtime/src/arena/tests.rs	Tests cover knob parsing, survival thresholds, policy state, retained-garbage limits, address-preserving promotion, old-generation indexing, fallback copying, and deferred registration behavior.
Release documentation and version metadata changelog.d/7744-gc-whole-block-in-place-promotion.md, CLAUDE.md, Cargo.toml	The changelog records implementation details, validation, benchmarks, and remaining costs. Version markers change to 0.5.1434.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related issues

• perf(gc): whole-block in-place promotion — kill the ~243 ns/object promotion tax on a fully-live nursery #7742 — Directly describes the whole-block in-place promotion implemented by this change.

Possibly related PRs

• PerryTS/perry#7624 — Its deferred old-page registration work connects to promoted-page registration.
• PerryTS/perry#7432 — Both modify copying-nursery promotion behavior, survival measurement, and GC telemetry.
• PerryTS/perry#7596 — Both modify promotion accounting and pacing for reclaimed or promoted capacity.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: whole-block in-place promotion for fully live young generations.
Description check	✅ Passed	The description provides detailed context, design notes, related issue, benchmarks, validation results, and remaining work, but it omits the template headings and checklist.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch gc/7742-page-promotion

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🧹 Nitpick comments (1)

crates/perry-runtime/src/gc/tests/promote_in_place.rs (1)

45-63: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Pin the intended promotion threshold.

Lines 49-60 accept every threshold from 5 through 999. A change from the measured boundary to 5 would pass this test and promote nearly every non-empty young generation. Assert the selected permille value, or test policy decisions on both adjacent boundary values.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/perry-runtime/src/gc/tests/promote_in_place.rs` around lines 45 - 63,
Strengthen threshold_separates_the_measured_workload_population by pinning the
intended PROMOTE_SURVIVAL_THRESHOLD_PERMILLE value or explicitly testing policy
behavior at both adjacent boundaries. Ensure the test fails if the threshold is
changed to an overly permissive value such as 5, while preserving the fully_live
and churny classification checks.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@changelog.d/7744-gc-whole-block-in-place-promotion.md`:
- Line 5: Update the retain.ts baseline value in the changelog benchmark summary
from 0.80 s to 0.81 s, matching the corresponding benchmark entry later in the
file and the PR description; leave the reported result and retain_wide.ts values
unchanged.

In `@crates/perry-runtime/src/arena/promote.rs`:
- Around line 36-38: Update the module documentation near
retag_young_for_in_place_promotion to state that it flips every in-use Eden
block and blocks from both survivor semispaces to generation Old in
HeapSpace::PromotedYoung. Keep the wording consistent with the implementation’s
unconditional [0usize, 1usize] iteration and the resulting no-Nursery invariant
used by skip_remembering.

In `@crates/perry-runtime/src/gc/copying.rs`:
- Around line 446-447: Correct the function reference in the documentation
comment near the copying optimization to use
debug_assert_no_remembering_possible, matching the defined and called function
name; leave the surrounding explanation unchanged.

In `@crates/perry-runtime/src/gc/tests/promote_in_place.rs`:
- Around line 84-90: Update an_unmeasured_thread_never_promotes to explicitly
exercise the absent-measurement state by adding a test helper that clears
LAST_YOUNG_SURVIVAL_PERMILLE and invoking it before
should_promote_young_in_place. Keep the assertion that promotion is rejected,
while ensuring the test no longer initializes the state with
InPlacePromotionTestGuard::enabled(0).

---

Nitpick comments:
In `@crates/perry-runtime/src/gc/tests/promote_in_place.rs`:
- Around line 45-63: Strengthen
threshold_separates_the_measured_workload_population by pinning the intended
PROMOTE_SURVIVAL_THRESHOLD_PERMILLE value or explicitly testing policy behavior
at both adjacent boundaries. Ensure the test fails if the threshold is changed
to an overly permissive value such as 5, while preserving the fully_live and
churny classification checks.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ef7cacb6-e9f7-4d27-a55a-bd14f254ceae

📥 Commits

Reviewing files that changed from the base of the PR and between caaab6b and dad34f3.

📒 Files selected for processing (14)

• changelog.d/7744-gc-whole-block-in-place-promotion.md
• crates/perry-runtime/src/arena/mod.rs
• crates/perry-runtime/src/arena/page_meta.rs
• crates/perry-runtime/src/arena/promote.rs
• crates/perry-runtime/src/arena/tests.rs
• crates/perry-runtime/src/gc/copying.rs
• crates/perry-runtime/src/gc/layout.rs
• crates/perry-runtime/src/gc/mod.rs
• crates/perry-runtime/src/gc/policy.rs
• crates/perry-runtime/src/gc/promote_in_place.rs
• crates/perry-runtime/src/gc/telemetry.rs
• crates/perry-runtime/src/gc/tests/mod.rs
• crates/perry-runtime/src/gc/tests/promote_in_place.rs
• scripts/addr_class_allowlist.txt

[proggeramlug]

Merging as v0.5.1434 — I reproduced the soundness evidence independently

The risk here is concentrated in one place: this skips three remembered-set passes on a promoting cycle. If the "after the retag no address classifies as Nursery" premise is ever wrong, a dropped remembered-set entry becomes a swept-live-object crash a cycle later — the worst failure shape this collector has.

So I built a retain-shaped workload of my own (3 M records, nothing ever dropped) and ran the instrument that speaks to exactly that:

PERRY_GC_FROMSPACE_SCAN_ABORT=1 PERRY_GC_DIAG=1 PERRY_GC_TRACE=1 ./n_retain
  → exit 0, sum 4499998500000 len 3000000
  → 5 cycles, each: missing_rewrites=0 dangling=0 never_dirty=0 lost_dirty=0 dirty_but_missed=0
  → promotion lines present — the new path was taken, not bypassed

never_dirty, lost_dirty and dirty_but_missed are precisely the "the remembered set is missing an edge" counters, and they are zero across every cycle with promotion active. That is the right instrument aimed at the right claim.

debug_assert_no_remembering_possible is also the right shape — it re-derives both premises from the heap (young bytes zero, malloc registry empty) rather than trusting the argument, and says so.

A/B on my own host (loaded dev box, so absolute numbers differ from the pinned mini, but same-host comparison is valid): 1.35 s → 0.57 s, corroborating the reported 0.81 → 0.53 in direction and magnitude.

What makes the design defensible rather than a gamble

• The decision is a measurement, and it self-corrects. Block liveness isn't knowable before the trace, so it's taken from the previous cycle's ratio — and a promoting cycle still traces, so it re-measures. A workload that flips from live to garbage pays one nursery of retained garbage and the policy turns itself off. The measured population being bimodal by three orders of magnitude (1.000 vs ≤0.004) is what makes a per-cycle carry-over safe.
• Old ⟹ TENURED (perf(gc): write barriers cost 16% on an all-numeric store workload — elide on provably-non-pointer stores #7511) upheld by stamping every header on a promoted block, dead ones included.
• Pacing was the hard part and is called out as such — the first working version was a 0.81 → 1.52 s regression, because promotion hands Eden's blocks away and the arena-bytes trigger's runway collapses. Returning the capacity as one-shot trigger headroom rather than re-reserved blocks avoids mapping memory the program may never touch (retain_wide peak RSS 470 → 447 MB).

The honesty I want to keep

Three limits stated rather than glossed:

• iso_miss.ts takes the ordinary copying path on all 50 minors, so the quarantine instrument's coverage is unchanged and does not cover the new path — which is why the from-space scan evidence exists separately.
• No gc_ratchet probe runs a promoting cycle, so the ratchet does not yet cover this either. Worth a probe.
• retain is still 4.4× Node, so perf(gc): whole-block in-place promotion — kill the ~243 ns/object promotion tax on a fully-live nursery #7742 stays open, with the residue named: the remembered-set scan over dirty element slots (26%) and one full collection reclaiming nothing (28%).

cargo test -p perry-runtime --lib: 1983 passed, 0 failed (one earlier run showed a single failure at host load 28; clean on re-run). Gates 21/21.