MultiStartGradient: Python-loop batched value_and_grad + jitted broad-start filter - #1431
Merged
Merged
Conversation
…-start filter Replace the in-XLA lax.map scan used when batch_size is set with a Python sweep over jit(vmap(value_and_grad)) chunks (ragged final chunk padded, so exactly one chunk-shaped program compiles per search), and jit the broad-start filter's single-point objective once per fit. The scan was compile-intractable (>55 min, OOM-killed) when its body is a multi-band FactorGraphModel fusion, and the eager filter cost a cache-immune ~13 min per multi-band process. Cold multi-band fit: intractable -> ~6.5 min (CPU and laptop GPU); warm ~2-3 min. Bit-identical numerics (parity including ragged chunks). PyAutoFit#1430.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Multi-band
af.FactorGraphModelfits driven byaf.MultiStartProdigypaid an intractable cold JAX compile (>1 h on CPU; the batchedjax.lax.mapscan is memory-explosive to compile and was OOM-killed even at batch 1) plus a cache-immune ~13 min eager broad-start filter in every process. This PR:lax.mapscan used whenbatch_sizeis set with a Python sweep overjit(jax.vmap(value_and_grad))chunks — the ragged final chunk is padded so exactly one(batch_size, ndim)-shaped program is compiled per search (pure_chunk_sliceshelper, numpy-only unit tested);_fitand reuses it across draws (it previously rebuilt an eagerjax.value_and_gradper draw — cache-immune, ~13 min per multi-band process).Production-path result (homogeneous 4-band jwst MGE
FactorGraphModel, 16 starts,batch_size=4; 1-core WSL host + RTX 2060): cold fit intractable → 395 s CPU / 392 s GPU; warm 136 s / 199 s.best_fombit-identical across every arm, backend and cache state. New GPU probe rows show the scan explosion is CPU-backend-specific and steady eval is at parity (0.740 vs 0.729 s per 16-start sweep), so no scan fallback is kept. Closes #1430's library leg; census rows + findings note land in the companion autolens_profiling PR.API Changes
None — internal changes only.
batch_sizesemantics are unchanged (numerically inert tiling; still the memory bound, now also the compile bound).See full details below.
Test Plan
python -m pytest test_autofit/— 1592 passed, 1 skipped (includes new numpy-only_chunk_slicestests: exact division, ragged, n < batch, batch=1, invariants)batch_size=Nonevs4vs ragged5— bit-identical best_fom / best_params / per-start params on the 1D-Gaussian JAX cellautofit_workspace_test/scripts/searches/MultiStartProdigy.pypasses unchanged (truth basin recovered)Full API Changes (for automation & release notes)
Removed
autofit.non_linear.search.mle.multi_start_gradient.search.jax_value_and_grad_single(fitness, vector)— module-level helper, never exported viaaf.*; superseded by a jitted single-point objective built inside_fit. No usages found in any workspace repo.Changed Behaviour
AbstractMultiStartGradient(batch_size=...)(allMultiStart*searches) — the chunked sweep now runs from a Python loop overjit(jax.vmap(value_and_grad))chunks instead of a singlejax.jit(lax.map(...))program. Numerically inert (bit-identical results); bounds both memory and XLA compile per chunk. Broad-start filtering now evaluates draws through one jitted (persistently cached) objective instead of eager per-draw AD — finite/non-finite classification of draws is unchanged in all validation runs.Generated by the PyAutoLabs agent workflow.