Skip to content

perf(gc): reuse discovery for internal reference counts - #1127

Merged
InauguralPhysicist merged 17 commits into
mainfrom
perf/gc-traversal-reuse-20260911
Sep 12, 2026
Merged

perf(gc): reuse discovery for internal reference counts#1127
InauguralPhysicist merged 17 commits into
mainfrom
perf/gc-traversal-reuse-20260911

Conversation

@InauguralPhysicist

@InauguralPhysicist InauguralPhysicist commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

The collector enumerated owning children again to count internal references after discovery. Count those references during discovery, and skip marking scans only for nodes proven to have no collector-node children in that collection. Duplicate edges, self-references, growth/rehash and module/environment/chunk ownership retain their existing semantics; the temporary child-presence array is freed on normal and accounting-abort paths.

The existing performance measurements belong to revision b63f9de, before CI integration. With identical generated EMS C linked against baseline 6fa1cb0 and that candidate, five-sample process-wall medians improved from 8.233209111 to 7.844329268 seconds for 4×4 odd Tseitin (4.72%) and from 12.267236136 to 11.559243344 seconds for 4×5 (5.77%). The comparison validated all 75 samples, eight UNSAT proofs, SAT models and unchanged search counters. These measurements have not been repeated for the integrated revision.

The focused C controls cover owning-edge multiplicity, live leaves, 600-node growth/rehash, and namespace/environment/chunk cycles. The positive control runs in sanitizer CI. Binary-swap selftests restore the original directory entry and inode, including when Bash 3 unwinds function locals before EXIT cleanup. Replay exit124 fails before success, boundary, or ledger classification; the suite requires all 11 replay controls.

This branch integrates the CI and macOS repairs from #1130 and the two action updates from #1125. The graphics gate uses bounded sanitizer probes and joined-worker leak controls; its early preflight has passed natively on Apple Silicon and Intel. The entire GC patch is unchanged by these harness integrations. Fresh combined validation:

  • Release and ASan+UBSan builds succeeded. Each build's collector controls run 2,455 assertions per variant: positive zero failures, duplicate-count fault five expected failures, disabled-skip fault one expected failure. ASan runs use detect_leaks=1.
  • Real Bash 3.2 and Bash 5.2: each 12 positive restoration controls and three original-defect rejections.
  • Replay selftest: all 11 cases; strict math 103/103 and executable-path regression 5/5 in both release and leak-enabled ASan; 260 unique suite labels.
  • Joined-worker sanitizer preflight: 2/2; timeout controls: 17/17. The final integrated head completed core ASan 5,079/5,079 and LSP 110/110 without sanitizer reports.
  • Independent collector, runtime, and gate reviews found no blocking source defect. Final CI on 7fc12c0: all 26 checks completed successfully, including Apple Silicon 5,072/5,072, Intel 5,075/5,075, and both sanitizer workers.

dependabot Bot and others added 2 commits September 9, 2026 08:27
Bumps the github-actions group with 2 updates: [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) and [actions/deploy-pages](https://github.com/actions/deploy-pages).


Updates `docker/setup-qemu-action` from 4.2.0 to 4.3.0
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](docker/setup-qemu-action@96fe6ef...1f40c72)

Updates `actions/deploy-pages` from 5.0.0 to 5.0.1
- [Release notes](https://github.com/actions/deploy-pages/releases)
- [Commits](actions/deploy-pages@cd2ce8f...368f825)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-version: 4.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
- dependency-name: actions/deploy-pages
  dependency-version: 5.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Count owning edges during discovery and cache per-collection child presence
to avoid redundant leaf marking scans. Preserve duplicate/self edges,
growth/rehash, module ownership, abort cleanup and the Value ABI.

Enroll focused sanitizer coverage and preserve strict test evidence across
binary swaps, LSan summary flushing and replay timeout classification.

Final gates: release5071/5071; ASAN5060/5060; focused2455 checks with
positive0, duplicatefault5, skip-disabled1 failures in release and ASAN.
Strict99 probes/31 pins/170 valid; failsoft174/174; JIT238x2/4 replay arms/
0ledger; replay238/12boundaries/0NONDET/0ledger; labels259 unique.
Binary-swap12 positives+3 reds; replay11 positives+5 guard-disabled reds.
Current-main compatibility52EMS+4DMG; certificate75samples/8DRAT proofs.
EMS n5:4x4 -4.72%,4x5 -5.77%, separated ranges; peakRSS119424KiB both.
DMG n5:249.890244918->249.890240718 instructions/emulated cycle;
wall3.110544634->3.155153577s (+1.43%); RSS6604->6600KiB.
Native<=2x goal remains open. RemoteCI still required (#1126).
Comment thread tools/bounded_process.py
can_raise=False
if session_members(process.pid):
raise RuntimeError('child exited with live session descendants')
except BaseException as exc:
Comment thread tools/bounded_process.py
if not members: return {'session':session,'signaled':sorted(signaled),'remaining':[]}
for pid in members:
try: os.kill(pid,signal.SIGSTOP)
except ProcessLookupError: pass
Comment thread tools/bounded_process.py
except ProcessLookupError: pass
for pid in session_members(session):
try: os.kill(pid,signal.SIGKILL); signaled.add(pid)
except ProcessLookupError: pass
Comment thread tools/bounded_process.py
try: cleanup=stop_session(process.pid)
finally:
try: process.wait(timeout=1)
except subprocess.TimeoutExpired: pass
Check portable sentinel substitution, bound owned-child teardown, and preserve
cleanup across repeated signals. Reject incomplete SIGUSR1 populations and
nonzero child exits. Support Bash 3 cleanup ownership and BSD wc padding.

Validation: 10 focused commands completed; SIGUSR1 default 15/15 and forced
13/13; strict 11/11; replay 6/6; portability 25/25; cleanup-deferral-disabled
control failed with the required delivered-signal/live-child witness.

Full remote release/ASan/macOS CI required before merge. Addresses #1126.
Run HTTP/model, gfx, zlib and net suites in separate fresh jobs with the
existing 30-minute limits. Preserve the current required extensions check
as an always-running aggregate that rejects every unsuccessful worker.

Validation: exact coverage of all 18 original non-checkout steps; dropped
and duplicated suite mutations rejected; 18 aggregate status cases passed.
Independent root rerun exit 0, chunk97b521. Actual four-worker CI pending.

Addresses #1126.
Bash 3 unwinds function-local scratch before the EXIT trap. Keep cleanup
state in the already-isolated subshell, and use a verified direct Python
child to exercise HUP/INT/TERM without the unavailable BASHPID variable.

Original helper failed first hardlink/guard case under real GNU Bash3.2;
xtrace observed empty scratch in the EXIT trap. Fixed full controls:
Bash3.2: 12 positives, 3 old-defect rejections, 9 signal witnesses.
Bash5.2: 12 positives, 3 old-defect rejections, 9 signal witnesses.
All 30 cases retained exact target/alias identities and expected cleanup.
Actual outer exits0 chunks f9b67a and b582c4. macOS CI still required.

Refs #1126, #1127.
@InauguralPhysicist

Copy link
Copy Markdown
Collaborator Author

The macOS restoration-control failure is reproduced under a pristine GNU Bash 3.2 build. Xtrace shows that Bash 3 unwinds the function-local scratch variable before the EXIT trap, so restoration sees an empty path and leaves the modified alias in place. Bash 5.2 passes the same original first case.

A follow-up fix is committed on codex/gc-swap-bash32-20260912 as 509ac64. It keeps cleanup state in the already-isolated subshell and replaces BASHPID-only signal injection with a verified direct child. Both Bash 3.2 and 5.2 passed 12 restoration cases and rejected all 3 original copy/move defect controls. Each shell recorded 9 actual signal deliveries; the 12 positive cases restored byte/inode/layout identity with no scratch residue. All case outputs and original/final identities are retained.

The original measurement worktree remains unchanged. This follow-up will be integrated with #1130 before final combined CI; neither this result nor the Linux reproduction establishes macOS CI success.

Assemble E000 before applying the existing UTF-8 message limit. Keep short
messages byte-identical and preserve the separate file-field behavior.
Cover short, long ASCII and multibyte missing paths with exact case enrollment.

Fixes #1132.

Validation: 10 bounded local commands; 3 original-path CLI checks and 3
dedicated cases; old-binary and missing-case faults both rejected. UTF-8 gate:
40 checks, 30 documented codes, 8 exemptions, 2000 identifier sweep runs,
1125 source-byte runs. All 9 original gate selftests passed. 1036 frozen
inputs unchanged. Full release/ASan CI remains required before merge.
Evidence: /tmp/ems-close-20260911/lint-e000-focused-r1/result.json
SHA256: 4aa4729cdaa784b41e155591acafee5d7728bf48a2808aed568a1147640c23d5
Show the first five explicit assertion lines and the last twenty captured
lines for failed EigenScript and lint children. Expected parser diagnostics
previously hid the later assertion. Consume the existing capture without
rerunning the child or changing its verdict, and preserve complete lines.
Fail setup if the formatter dependency is missing.

Validation: formatter 3 positive cases and 2 attributed fault rejections;
shell syntax and git diff checks passed. Full suite execution remains CI work.
Evidence: /tmp/ems-close-20260911/failure-output-controls/result-r1.json
SHA256: 326c1e3c0c9e06411d49740461fc39e4bf68d8c62ea717219a7466f59dd98282
Core ASan plus LSP and HTTP/model ASan now own separate checkouts and retain
the original 45-minute limits. Keep the existing required status as an always
aggregate that requires both workers and a successful scope check.

The prior job passed core 5060/5060 and LSP 110/110, then exhausted its
cumulative budget during the second suite. No checks or sanitizer options
are removed, and no incomplete suite is accepted.

Validation: exact preservation of all 4 original non-checkout steps; dropped
and duplicated suite controls rejected; 15 actual aggregate-shell cases passed,
including rejection of failure, cancellation, skipped or missing workers and
unknown scope output. Full remote completion remains required.
Evidence: /tmp/ems-close-20260911/sanitizer-split-review/result-r1.json
SHA256: d663ca4fdd2925f908cebddec29c691de53fc289a02ad4be4d50fa21fafffd4f
Resolve and own the executable path before chdir, use separate rounded
matrix multiply/add operations, and preserve the pinned buffer NaN result.
Use upstream LLVM for macOS gfx leak checks and retain invalid-path lint
coverage on filesystems that reject invalid UTF-8 names.

Measured gate results:
EXE PATH: 5 passed, 0 failed (native and no-proc); old no-proc: 1/5.
STRICT: 103 passed, 0 failed.
identical-when-off: 99 differing: 0
raises-under-strict: 99 silent: 0 misattributed: 0
answer-pins held: 31 broken: 0
valid-input rows unchanged in BOTH modes: 170 / 170
UTF-8: 40 checks, 30 documented codes, 8 pinned exemptions;
2000 identifier runs and 1125 source-byte runs.
Results: 247 passed, 0 failed, 247 total (lint).
Doc examples: 88 checked, 88 passed, 0 failed, 5 skipped.
Changed compiler audit: 4/4 and recognizer coverage passed;
compiler-audit integration and synthetic fault selftests: OK.
gfx startup controls: 2/2; original wrongly 15/15, fixed rejects startup.
gfx symbol controls: 3/3; independent corpus decisions: 20/20.
Independent runtime and gate reviews: PASS within local scope.

Full integrated platform and sanitizer CI pending; no native macOS pass
is claimed from the local controls.
Compared with bb49292, the integrated tree differs only in the two intended action pins (2 files, 2 substitutions). Runtime sources and test harnesses are identical to that locally verified repair. Full integrated CI remains required before landing.
Preserve the collector ownership check after the core sanitizer build,
all 11 replay controls and exit124 rejection, and the Bash 3 restoration fix.

Combined release build: exit 0.
positive: gc-traversal: cases=3 checks=2455 failures=0
duplicate-count: gc-traversal: cases=3 checks=2455 failures=5
skip-disabled: gc-traversal: cases=3 checks=2455 failures=1
binary-swap-controls: positives=12 old-defect-rejections=3
(real Bash 3.2 and Bash 5.2).
SELFTEST: all planted faults caught (11 replay cases).
STRICT: 103 passed, 0 failed
EXE PATH: 5 passed, 0 failed
PASS: 260 labelled echo lines, no two sections share a label

Full integrated platform and sanitizer CI remains required before landing.
Comment thread src/builtins_tensor.c
* changing buffer-NaN reads generally is a separate reform tracked
* in ROADMAP.md. Finite results and infinities remain untouched. */
for (int i = 0; i < res->data.buffer.count; i++) {
if (res->data.buffer.data[i] != res->data.buffer.data[i]) {
The Apple Silicon job hung for 28 minutes in the tiny ASan startup
control with Homebrew LLVM 20.1.8. Select LLVM 22, whose released runtime
contains llvm-project#182943 via the release/22.x backport #188913.
Keep the interpreter on Apple Clang and retain the deliberate leak control.

Bound sanitizer children with TERM after 120s and KILL after 5s, including
the startup and SDL probes. Require a completed positive leak control;
a timeout after leak output is a failure. Missing coreutils fails loudly.

Give the full native job 45 minutes: Intel completed the gfx gate and
540 compiler invocations but hit 30 minutes in compiler audit selftests.
No test populations or required checks are removed.

Validation: timeout controls 17 passed, 0 failed under Bash 5 and Bash 3.2;
5 private timeout/exit mutations rejected, 0 missed. Changed compiler
audit 4/4 invocations; recognizer coverage 102 scripts, 7 shape waivers.
Workflow parse/native coverage, bash syntax and diff checks are green.
Native macOS CI must verify the selected runtime and integrated suite.
Integrate 597b287: bounded gfx sanitizer startup, fixed LLVM22 on macOS26,
and measured native suite budget. Runtime and gate files are byte-identical
to that reviewed head; only the two intended Docker/Pages action pins differ.
Inherited verification: 17/17 timeout controls on Bash5 and Bash3.2, five
mutations rejected, changed compiler audit4/4 and recognizer102scripts green.
The integrated PR CI must complete before merge.
Integrate 63ec630/597b287 without changing GC/runtime source or the two
action updates. The timeout files match reviewed 597b287; the complete GC
patch relative to the dependency branch remains unchanged by integration.

Fresh integrated timeout controls: 17 passed, 0 failed. Source and Makefile
match e35805c, whose core ASan CI completed 5079/5079 and LSP 110/110 with
no sanitizer report. Full native/integrated CI remains required before merge.
Comment thread tests/test_gfx_timeout.py
if pidfile.exists():
try:
os.killpg(os.getpgid(int(pidfile.read_text())), signal.SIGKILL)
except ProcessLookupError:
LLVM22 fixed the macOS26 ARM startup hang, but the original -O0 leak
fixture exited 0 without a report. Allocate inside a joined worker which
returns NULL so the main thread never retains an allocation pointer. The
clean twin differs only by freeing that allocation. Scanner settings stay
unchanged, and thread/allocation failures cannot pass either control.

Add --toolchain-only to the same gate and run it as an early native CI
step. It validates startup and both leak controls before any gfx build,
fails when ASan is unavailable, and does not replace the full suite.

Validation: real GCC preflight 2/2; timeout controls 17/17; invalid arguments
rejected 3/3; independent review PASS. Full graphics population remains 15.
Changed compiler audit 4/4; recognizer 102 scripts, 7 shape waivers. Workflow
preflight ordering, shell syntax and diff checks are green.
Native macOS must verify the real joined-thread controls and full suite.
Integrate 2c30282 with joined-worker leak/clean controls and early native
preflight. Only the two intended Docker/Pages action pins differ from that
reviewed head. Its actual GCC preflight passed 2/2; timeout controls 17/17,
invalid arguments 3/3 rejected, compiler audit 4/4 and recognizer 102 scripts
passed. Final integrated CI remains required before merging.
Integrate f75a2bf/2c30282. The complete GC patch relative to the dependency
branch is unchanged. Runtime source and Makefile match 1617fb3; actual gfx
control sources match reviewed 2c30282. Its real GCC preflight passed 2/2,
timeout controls 17/17, invalid args rejected 3/3, and compiler audit 4/4 with
recognizer coverage 102 scripts. Full native and integrated CI remains
required before merging.
@InauguralPhysicist
InauguralPhysicist merged commit 7b4302e into main Sep 12, 2026
26 checks passed
@InauguralPhysicist
InauguralPhysicist deleted the perf/gc-traversal-reuse-20260911 branch September 12, 2026 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants