fix: pin fork start method for pools (Python 3.14 forkserver default)

[Jammy2211]

Overview

Python 3.14 support is blocked by a single reproducible failure: the factor-graph tutorial (autofit_workspace/scripts/overview/overview_1_the_basics.py) dies with TypeError: 'Gaussian' object is not iterable inside a dynesty pool worker. Root cause is now confirmed empirically: Python 3.14 changed the default multiprocessing start method on Linux from fork to forkserver (workspace scripts are unguarded and forkserver preloads __main__, so worker state is re-derived/pickled instead of fork-inherited). Forcing the fork start method makes the identical script complete cleanly on 3.14.4. The fix is to pin an explicit fork context at PyAutoFit's pool-creation sites on POSIX, restoring pre-3.14 semantics, then re-add 3.14 to the evidence matrix and classifiers.

Plan

• Add one central fork-context helper to PyAutoFit's parallel machinery (POSIX-only; platform default retained on Windows).
• Use it at every pool/process creation site: dynesty (via a small Pool subclass, since upstream hardcodes mp.Pool), make_pool, SneakyPool, the raw Process/Queue layer, EP optimiser, and pass nautilus a fork-context pool object instead of an int.
• Add a numpy-only unit test asserting autofit pools use the fork context on POSIX.
• Verify the failing tutorial passes on a Python 3.14 venv, and the suite stays green on 3.12/3.13.
• Follow-up (separate small PRs once this merges): re-add 3.14 to PyAutoHands python_matrix.yml and library classifiers; retarget the PyAutoNerves 3.14 experimental banner.

Detailed implementation plan

Affected Repositories

• PyAutoFit (primary — this PR)
• PyAutoHands + library classifiers + PyAutoNerves banner (follow-up PRs, gated on this fix)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoFit	main (== origin/main, 09a134d)	clean

Three existing worktrees (point-source-defaults-campaign, potential-correction-validation, start-here-other-feature-sections) contain PyAutoFit but all sit on undiverged main — no conflict.

Suggested branch: feature/py314-fork-context

Root-cause evidence (py3.14.4 venv, full editable stack)

• Full dependency stack installs cleanly on 3.14 (jax 0.10.2, numba 0.66, numpy 2.4.6, dynesty 2.1.5 all have wheels) — no dependency blocker.
• In-process factor-graph likelihood and pickle round-trips of FactorGraphModel / global prior model are correct on 3.14 — the prompt's hypotheses 1 (iteration protocol) and 2 (model flattening) are ruled out.
• Unmodified tutorial under the 3.14 default (forkserver): reproduces the exact CI failure ('Gaussian' object is not iterable raised through dynesty/pool.py map).
• Identical tutorial with multiprocessing.set_start_method("fork", force=True) (runpy wrapper): completes end-to-end.
• A __main__-guarded replication of autofit's exact dynesty Pool(...) call works under both fork and forkserver — the breakage is the unguarded-script + forkserver combination, and workspace scripts are unguarded by design.
• Second silent failure mode: on 3.14 the dynesty pool creation can raise RuntimeError, which abstract.py swallows into the misleading "Your operating system does not support Python multiprocessing" single-CPU fallback — scripts that "pass" silently lose parallelism.

Implementation Steps

1. New helper in autofit/non_linear/parallel/ (e.g. context.py): mp_context() returning multiprocessing.get_context("fork") on POSIX and the default context on Windows. One place to document the 3.14 rationale.
2. autofit/non_linear/search/nest/dynesty/search/abstract.py:214 — dynesty's Pool.__enter__ hardcodes mp.Pool(...); add a small subclass overriding __enter__ to build the pool from the fork context (identical initializer/initargs), and use it in the with Pool(...) block.
3. autofit/non_linear/search/abstract_search.py:1319 (make_pool) — mp.Pool(...) → fork-context pool.
4. autofit/non_linear/parallel/sneaky.py:394 (SneakyPool, covers emcee/zeus) — mp.Pool(...) → fork-context pool.
5. autofit/non_linear/parallel/process.py — multiprocessing.Process/Queue → fork-context equivalents.
6. autofit/graphical/expectation_propagation/optimiser.py:463 — multiprocessing.Pool(...) → fork-context pool.
7. autofit/non_linear/search/nest/nautilus/search.py:331 — nautilus builds its own default-context pool when passed pool=int; pass a fork-context pool object instead (nautilus accepts pool objects).
8. Unit test in test_autofit/ (numpy-only, no JAX): assert pools produced by the helper / make_pool use the fork context on POSIX; skip on Windows.
9. Validation: full pytest on 3.12/3.13; on a 3.14 venv run the reproducer from the prompt (PYAUTO_TEST_MODE=1 PYAUTO_SMALL_DATASETS=1 python3.14 autofit_workspace/scripts/overview/overview_1_the_basics.py) and confirm clean completion.

Key Files

• autofit/non_linear/parallel/context.py — new fork-context helper (name TBD)
• autofit/non_linear/search/nest/dynesty/search/abstract.py — dynesty pool subclass + use
• autofit/non_linear/search/abstract_search.py — make_pool
• autofit/non_linear/parallel/sneaky.py, autofit/non_linear/parallel/process.py — SneakyPool / Process / Queue
• autofit/graphical/expectation_propagation/optimiser.py — EP pool
• autofit/non_linear/search/nest/nautilus/search.py — pass pool object

Trade-offs / risks

• CPython warns on fork-with-threads (since 3.12); pinning fork is the pragmatic fix that preserves long-standing semantics — guarding every workspace script is not viable. Revisit if CPython restricts fork further.
• fork context is unavailable on Windows: helper falls back to the platform default there (spawn), which is today's behavior.
• The micro-mechanism of the instance flattening inside forkserver workers was not fully traced; the fix is justified by restoring the exact pre-3.14 execution model, verified end-to-end.

Original Prompt

Click to expand starting prompt

Investigate FactorGraphModel instance shape on Python 3.14

Type: bug
Target: PyAutoFit
Difficulty: too-large
Autonomy: supervised
Priority: normal
Status: formalised

Problem

autofit_workspace/scripts/overview/overview_1_the_basics.py runs cleanly
on Python 3.9–3.13 but fails on Python 3.14 with:

TypeError: 'Gaussian' object is not iterable

This was surfaced by the python_matrix.yml evidence run (issue
PyAutoLabs/PyAutoBuild#74, run 25208343008). Same script, same library
code — only the Python version differs.

3.14 was dropped from advertised support (classifiers + python_matrix.yml
matrix) until this is understood. requires-python still allows >=3.9,
so users who install on 3.14 anyway will see the import-time banner warn
that 3.14 isn't first-class.

Reproducer

Run on Python 3.14 in a venv with the libraries installed editable:

python3.14 -m venv /tmp/py314
/tmp/py314/bin/pip install -e PyAutoConf -e PyAutoArray -e PyAutoFit \
    -e PyAutoGalaxy -e "PyAutoLens[optional]"
PYAUTO_TEST_MODE=1 PYAUTO_SMALL_DATASETS=1 \
    /tmp/py314/bin/python autofit_workspace/scripts/overview/overview_1_the_basics.py

Reproduces immediately when the script reaches the factor_graph = af.FactorGraphModel(*analysis_factor_list) block (~line 545).

Stack trace (abridged)

File "autofit_workspace/scripts/overview/overview_1_the_basics.py", line 813
    [profile_1d.model_data_from(xvalues=xvalues) for profile_1d in instance]
                                                                   ^^^^^^^^
TypeError: 'Gaussian' object is not iterable

The above exception was the direct cause of the following exception:

File "autofit_workspace/scripts/overview/overview_1_the_basics.py", line 563
    result_list = search.fit(model=factor_graph.global_prior_model,
                             analysis=factor_graph)
File "PyAutoFit/autofit/non_linear/search/abstract_search.py", line 668
    search_internal, fitness = self._fit(...)
File "PyAutoFit/autofit/graphical/declarative/collection.py", line 105
    log_likelihood += model_factor.log_likelihood_function(instance_)
File "PyAutoFit/autofit/graphical/declarative/factor/analysis.py", line 189
    return self.analysis.log_likelihood_function(instance)

The workspace Analysis.log_likelihood_function expects instance to be
a Collection of profiles (since the model was built as
af.Collection(gaussian=Gaussian(), exponential=Exponential())). On
3.9–3.13 it receives a Collection; on 3.14 it receives a single
Gaussian object directly.

What we know

• model = af.Collection(gaussian=Gaussian(), exponential=Exponential())
• Each AnalysisFactor is built with model.copy() — so each factor's
  prior_model is itself a Collection of two profiles.
• FactorGraphModel(*analysis_factor_list).global_prior_model is the
  combined model passed to the search.
• On 3.9–3.13: zip(self.model_factors, instance) in
  collection.py:104 yields (factor, sub_instance) pairs where
  sub_instance is the per-factor Collection instance — iterable.
• On 3.14: the same iteration yields a single Gaussian instance — not
  iterable.

So either:

1. ModelInstance.__iter__ (which falls back to __getitem__ since
   ModelInstance has no explicit __iter__) yields different child
   types on 3.14, OR
2. FactorGraphModel.global_prior_model constructs a flatter structure
   on 3.14 (collapses nested Collections into scalars), OR
3. dynesty's multiprocessing pickling round-trips the model differently
   on 3.14 (the trace shows multiprocessing.pool.RemoteTraceback,
   suggesting the worker process saw a different structure than the
   main process).

Where to start investigating

1. Print instance and type(instance) at the top of the workspace
   Analysis.log_likelihood_function on a 3.14 venv. Compare against
   3.13. Specifically:

   • Is instance a ModelInstance, a Collection, or a raw
     Gaussian?
   • What does instance.__dict__ look like on each version?
   • What does list(instance) do?

2. autofit/mapper/model.py:385 ModelInstance has no explicit
   __iter__ — Python falls back to the legacy sequence protocol via
   __getitem__. ModelInstance.__getitem__(int) returns
   list(self.values())[item]. On 3.14, check whether values() and
   the resulting iteration yield different types than on 3.13.

3. FactorGraphModel.global_prior_model — trace how the per-factor
   Collection structures get composed into the global model. If 3.14
   flattens Collection -> [profile_1, profile_2] into bare profiles
   (because the dict ordering or attribute lookup behaves differently),
   the per-factor instance_ would be a single profile.

4. Check whether dynesty.pool (from the failure trace) round-trips
   the model object correctly on 3.14. The error is wrapped in a
   multiprocessing.pool.RemoteTraceback, so the failure is happening
   in a worker process. Try dynesty(parallel=False) to isolate.

5. Python 3.14 release notes worth scanning for relevant behavior
   changes:

   • PEP 768: safe external debugger interface
   • PEP 749: late-bound default values (annotations)
   • PEP 765: disallow return/break/continue in finally
   • Behavior changes around dict ordering, __init_subclass__,
     __set_name__, descriptor lookup

Constraints when fixing

• Don't modify the workspace tutorial script just to paper over the
  symptom. The script worked on 3.9–3.13 because the library produced
  the right shape; it should produce that same shape on 3.14.
• Library unit tests must remain numpy-only — don't add jax-dependent
  tests for this.
• If the fix requires a workspace-side change too (e.g. a defensive
  helper), keep it minimal and add a comment pointing back to this
  prompt.

Done when

• python3.14 autofit_workspace/scripts/overview/overview_1_the_basics.py
  runs cleanly under PYAUTO_TEST_MODE=1.
• 3.14 can be re-added to PyAutoBuild's python_matrix.yml matrix and
  to each library's pyproject.toml classifiers.
• The change is unit-tested in test_autofit/ with a numpy-only
  factor-graph round-trip test that would have caught the 3.14 shape
  collapse on 3.13 too if the structure had been wrong there.

[Jammy2211]

Library PR created + smoke sweep

PR: #1439 (feature/py314-fork-context, pending-release) — opened under the human-authorized Heart-RED override (RED = the unrelated nightly "release validation FAILED (stage integrate)").

Validation summary

• Full test_autofit in the worktree: 1641 passed, 2 skipped, incl. new numpy-only fork-context tests (factor-graph likelihood computed in a real fork-pool worker == in-process value).
• Python 3.14.4 venv, default start method: overview_1_the_basics.py completes end-to-end (previously failed at the factor-graph fit with 'Gaussian' object is not iterable). Unmodified library reproduces the failure — single-variable confirmation of the forkserver root cause.

Six-workspace smoke sweep (worktree autofit asserted via autofit.__file__):

workspace	passed	failed	skipped
autofit_workspace	7	0	1
autogalaxy_workspace	13	0	0
autolens_workspace	34	0	1
autolens_workspace_test	8	11	2
euclid_strong_lens_modeling_pipeline	6	0	0
HowToLens	6	0	0

All 11 failures are in autolens_workspace_test and are pre-existing, not attributable to this PR. Controls re-run sequentially against unchanged main autofit fail bit-identically:

• imaging/jax_likelihood/mge.py — same vmap mismatch (ACTUAL -4664.718875 / DESIRED -86629.349379) on main and on the branch (cluster: 6 jax_likelihood vmap-mismatch scripts).
• multi_galaxy/composition_mge.py — same prior_count: expected 6, got 4 on main.
• imaging/jax_likelihood/lp.py — fails on main.
  The remaining failures are the potential-correction dpsi ValueErrors (in-flight feature/models with type #672 wst work) and a stochastic subhalo-recovery threshold. wst is also 2 commits behind origin/main. These belong to the wst owners' in-flight tasks, not this change.

Also flagged during validation (pre-existing, reproduces on unchanged main, py3.13): multi-core emcee (number_of_cores=2) hangs on the WSL dev box while single-core completes instantly — worth a separate look.

Remaining for 3.14 promotion after this merges (separate PRs): re-add 3.14 to PyAutoHands python_matrix.yml, add 3.14 classifiers to the libraries, retarget the PyAutoNerves 3.14 experimental banner.

[Jammy2211]

Shipped: PR #1439 merged to main (3467de94a), CI green on 3.12/3.13. Completion record: PyAutoMind/complete/2026/07/py314-fork-context.md. Python 3.14 promotion follow-ups (matrix re-add, classifiers, Nerves banner) tracked there as next small PRs.