The SingleStepTests suites (pysnes/cpu/test_cpu.py, pysnes/apu/test_spc700.py)
allocate roughly 1.2 GB of anonymous memory per xdist worker, so the
invocation currently recommended in AGENTS.md —
uv run pytest -n auto --dist=loadgroup
— scales linearly with core count and will OOM the machine. On a 16-core
host it hard-crashed the desktop twice (global OOM, the kernel started
killing unrelated processes). Kernel task dump from one of the crashes:
python/pypy procs with >200MB anon: 23, combined: 24.8 GB
on a 31 GB machine that already had ~8.6 GB of unrelated resident load.
Why it happens
Each worker parses SingleStepTests JSON files and holds the decoded cases
in memory. --dist=loadgroup pins all cases from one JSON file to a single
worker, and _load_case keeps a single-slot file cache — good for speed,
but it means every worker independently holds a fully decoded JSON file
(these are large), and nothing bounds the total.
What to change
- Cap memory per worker instead of letting a decoded file sit resident —
e.g. stream cases with ijson (already a dependency) rather than
decoding whole files, or store the collection cache in a compact binary
form workers can read incrementally.
- Stop recommending
-n auto for these suites. Derive a default worker
count from available memory, not core count, and document a safe
explicit value.
- Consider putting the memory-heavy suites behind a marker, like the
existing integration marker, so a plain pytest run stays light.
Docs to update once fixed
AGENTS.md currently states "Recommended invocation: -n auto --dist=loadgroup" for test_cpu.py and test_spc700.py. That advice
needs to change alongside the fix.
The SingleStepTests suites (
pysnes/cpu/test_cpu.py,pysnes/apu/test_spc700.py)allocate roughly 1.2 GB of anonymous memory per xdist worker, so the
invocation currently recommended in
AGENTS.md—— scales linearly with core count and will OOM the machine. On a 16-core
host it hard-crashed the desktop twice (global OOM, the kernel started
killing unrelated processes). Kernel task dump from one of the crashes:
on a 31 GB machine that already had ~8.6 GB of unrelated resident load.
Why it happens
Each worker parses SingleStepTests JSON files and holds the decoded cases
in memory.
--dist=loadgrouppins all cases from one JSON file to a singleworker, and
_load_casekeeps a single-slot file cache — good for speed,but it means every worker independently holds a fully decoded JSON file
(these are large), and nothing bounds the total.
What to change
e.g. stream cases with
ijson(already a dependency) rather thandecoding whole files, or store the collection cache in a compact binary
form workers can read incrementally.
-n autofor these suites. Derive a default workercount from available memory, not core count, and document a safe
explicit value.
existing
integrationmarker, so a plainpytestrun stays light.Docs to update once fixed
AGENTS.mdcurrently states "Recommended invocation:-n auto --dist=loadgroup" fortest_cpu.pyandtest_spc700.py. That adviceneeds to change alongside the fix.