results: #678 phase B evidence campaign — truth-anchored A100 cells + campaign notes - #100
Merged
Merged
Conversation
…discriminator cells, cluster searches, A100 submits - Truth-anchored methodology: simulators write truth.json; every point_source/ cluster search JSON now records results.truth_log_likelihood and delta_max_ll_vs_truth (evaluated with the cell's own fit class at the simulator truth), plus results.posterior_stats (per-param mean/std) for posterior-width comparisons. - New model types: source_plane_tensor (free-centre weighting='jacobian', PyAutoLens#679) and image_plane_repeat_solved; six new galaxy leaf cells. - Instrument variants for the pairing discriminator + domain-of-validity: simple_missing (image dropped), simple_extra (spurious position), near_caustic (source at 0.95x tangential caustic). - Cluster tier searches: dataset_class='cluster' (factor-graph AnalysisPoint per system, cosmology pinned — solver custom_jvp constraint), ported model on the repo's dPIEMassB0Sph simulator; seven cluster leaf cells. - FIX cluster simulator: solver.solve was called without plane_redshift, so every source's positions were solved to the FINAL plane — the z=1.0 source's committed positions were unphysical (truth solved logL -4.2e6; +26.1 after the fix). Same bug exists in autolens_workspace cluster/simulator.py (flagged on autolens_workspace#436). - 25 A100 submit scripts + submit_all_678_phase_b.sh driver (pre-generates datasets before sbatch to avoid the concurrent auto-simulate race).
…n A100) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnTmLoJjJgMTze5uAbg1Jd
…mpaign notes Tensor ranks truth first at both tiers; free-centre plateau survives logsumexp (solved confirmed); missing-image arm decisive for PairAll (PairRepeatSolved mis-ranks truth by 1.8e5 logL); solved not overconfident (exp-4); tensor holds at 0.95x caustic (exp-5). In flight: simple_extra 8h reruns (331885/331886) + msp cluster image_plane_solved rerun (331887) on the merged gradient fixes (PyAutoFit#1441, PyAutoLens#685). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnTmLoJjJgMTze5uAbg1Jd
…ean free-cell re-runs Closes out the phase-B evidence campaign with the last three A100 jobs. 331887 (multi_start_prodigy/cluster/image_plane_solved): the cell that exposed PyAutoFit#1441 and PyAutoLens#685 now completes end-to-end on the A100 — 832 s, finite value, no NaN — confirming both merged fixes hold through a full search and not just the 8-draw gradient probe. Its truth anchor (+14.658) matches the Nautilus twin's exactly, cross-validating the run. But the search still misses the basin (delta -1723.6 vs Nautilus +16.9). This was the last candidate for a cluster-scale gradient-search recommendation, and it fails alongside the source-plane cells (-14.8, -11062). The notes' caveat is corrected accordingly: at cluster scale every objective tested is Nautilus's. Making the gradients finite was a correctness fix; it did not buy convergence. A negative delta is an optimizer defect, not a likelihood one, so none of this touches the choice of default. 331888/331889 replace the two free `simple` Nautilus cells that had been silent PyAutoFit resumes (sub-second sampler wall). Both reproduce their prior conclusions within scatter — image-plane delta +2.27 -> +2.38, source-plane +33476.5 -> +33474.5 against an identical truth anchor — so the resumes had not misled, but the shipped numbers now rest on searches that actually ran. exp-4's free einstein_radius std moves 0.0273 -> 0.0293 (solved bars ~28% wider, not ~40%; direction unchanged). The two simple_extra exp-3 arms are recorded as a deliberate non-result: both timed out at the 8h wall (331885/331886), f_live 0.92 and N_eff 12 on the all-to-all arm. A spurious position imposes an honest ~-1.4e4 floor on every model, so Nautilus cannot converge. The DNF is symmetric, so this arm does not discriminate between the pairings — the missing-image arm is what the default rests on, and it was already in hand. Also records two harness quirks that are easy to misread as defects of this run: likelihood_evals is 65 for every MultiStartProdigy cluster cell (it counts vmapped batch calls), and best_fit is an AttributeError on every cluster cell including the Nautilus ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The phase-B plane_redshift fix moved jitted_solve from a module-level function into a per-iteration assignment from jitted_solve_for(), which made it a loop variable that the jit_profile lambda captured by reference. Bound as a default argument, matching the _tracer=tracer idiom already on the same line. Not a live bug — jit_profile consumes the lambda inside the same iteration, so the late binding never resolved to the wrong solve — but ruff B023 is right that it is fragile, and this is what turned lint red on the branch. Verified against main's unchanged file as a control: main passes B023, the branch failed, so the branch introduced it rather than a ruff-version change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pure line-wrapping to satisfy the lint workflow's 'ruff format --check' step at the repo's line-length = 100. No semantic change. These are phase-B harness files, not the results commit. The branch had been pushed since 2026-08-01 with no PR open, so neither ruff step had ever run against it. Reproduced locally first — the same 3 files, byte-identical diffs — so this is the repo's own formatter and not a CI-only ruff version difference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ange The range sits directly under a table whose first column is -33788.4, so it reads as though all six source-plane cells fall in it. Five do; the galaxy free-scalar cell is at +33474.5. Per the doc's own reading key that is the LIKELIHOOD mis-ranking, not a search failure — Nautilus found its objective's best basin fine, the basin is simply nowhere near truth. Stated explicitly so the bias showcase is not misread as a Nautilus result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cross-review of the open PRs (7 findings, all verified before acting). MECHANISM WAS WRONG (F2). Both docs said repeat pairing 'has no way to leave a model-predicted image unmatched'. It does — pair_repeat.py builds an unmatched_model_mask and penalizes through unmatched_model_policy. Checked the source rather than reasoning about it. The real mechanism: with an observed image missing, the model image that would have paired to it is the nearest neighbour of no observed position, so the default magnification_filter policy sees a BRIGHT extra image (the exemption is for demagnified ones) and charges its full distance to the nearest other observed position. Truth is punished for correctly predicting an image the data lacks. Symptom and conclusion unchanged; the explanation a reader would learn from was wrong. UNSUPPORTED EVIDENCE (F3). The f_live=0.92 / N_eff=12 / logZ=-14238 figures for the DNF arms are NOT reproducible from this PR. Went looking on RAL: the surviving output trees hold only metadata, model.info, .identifier and a 12-line search.log with no sampler statistics, no Nautilus checkpoint, and no SLURM stdout anywhere under /mnt/ral/jnightin. So the diagnostic is recorded interpretation of a run whose output is gone, now labelled as such. What IS reproducible is stated separately: sacct still reports both TIMEOUT, and the clean siblings' JSONs give the 147.2 s / 162.7 s comparison walls. Also concedes the epistemic point that a timeout proves non-completion within the cap, not destroyed convergence. UNATTRIBUTED PROBE (F4). '8/8 random cluster draws' has no artifact in this repo; now points at the bug-fix PRs and the autofit_workspace_test#81 regression test as the citable form. SELF-CONTRADICTION (F1). The note called solved+scalar a preference for wrong models at delta +5.8, which its own reading key calls ordinary basin recovery. Corrected to what the numbers show: the catastrophic scalar mis-ranking is a FREE-CENTRE effect, solving the centre already removes most of it, and the tensor is the further refinement that puts truth highest (+12.75 / +0.60 / -33788). WRONG WALL (F5). Nautilus cluster twin is 12.4 min total (742.08 s), not 12.6. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The autolens_profiling half of the point-source defaults campaign
(PyAutoLens#678): the
evidence-campaign harness, 23 truth-anchored A100 result cells, and the written
synthesis the workspace guides cite.
Every cell evaluates its likelihood flavour at the simulator-truth model and
scores the search by
delta = max_log_likelihood − truth_log_likelihood, so alikelihood defect (a wrong model out-ranking truth) is distinguishable from a
search defect (failure to reach the basin). That distinction is what the
campaign's conclusions rest on.
What the evidence established
PairRepeatSolvedmis-ranks truth by +1.8e5 whilePairAllSolvedrecovers it(+1.3). On clean data the two are equivalent, so robustness decides. This is
what shipped as PyAutoLens#686.
convention mis-maps the μ=367 image's radial noise.
as needing sampler muscle (the free-centre gradient plateau survives the
phase-A logsumexp fix).
PyAutoFit#1441 (traced attributes classified as pytree aux) and PyAutoLens#685
(padded solver rows anchored at NaN-Jacobian centres). Both merged.
Evidence tail in this PR
The final commit closes the campaign out with the last three A100 jobs:
defects. It now completes end-to-end (832 s, finite, no NaN), validating the
merged fixes through a full search. But it still misses the basin
(delta −1723.6 vs Nautilus +16.9), which closes the one question the notes had
left open: the solved image-plane cell was the last candidate for a
cluster-scale gradient-search recommendation, and it fails alongside the
source-plane cells. At cluster scale, every objective tested is Nautilus's.
Finite gradients were a correctness fix; they did not buy convergence.
(sub-second sampler wall — identifiers ignore the dataset). Both reproduce
their prior conclusions within scatter, so the resumes had not misled, but the
shipped numbers now rest on searches that actually ran.
simple_extraarms timed out at 8h walls. A spurious position imposes an honest ~−1.4e4
floor on every model, so Nautilus never converges. The DNF is symmetric,
so this arm does not discriminate between the pairings — the default rests on
the missing-image arm, which was already in hand.
Notes
of a given run:
likelihood_evalsis 65 for every MultiStartProdigy clustercell (it counts vmapped batch calls), and
best_fitis anAttributeErroronevery cluster cell including the Nautilus ones.
prose that cites these numbers is autolens_workspace#468.
🤖 Generated with Claude Code