Skip to content

results: #678 phase B evidence campaign — truth-anchored A100 cells + campaign notes - #100

Merged
Jammy2211 merged 8 commits into
mainfrom
feature/point-source-defaults-campaign
Aug 4, 2026
Merged

results: #678 phase B evidence campaign — truth-anchored A100 cells + campaign notes#100
Jammy2211 merged 8 commits into
mainfrom
feature/point-source-defaults-campaign

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

The autolens_profiling half of the point-source defaults campaign
(PyAutoLens#678): the
evidence-campaign harness, 23 truth-anchored A100 result cells, and the written
synthesis the workspace guides cite.

Every cell evaluates its likelihood flavour at the simulator-truth model and
scores the search by delta = max_log_likelihood − truth_log_likelihood, so a
likelihood defect (a wrong model out-ranking truth) is distinguishable from a
search defect (failure to reach the basin). That distinction is what the
campaign's conclusions rest on.

What the evidence established

  • All-to-all pairing as the default — on a quad with one true image removed,
    PairRepeatSolved mis-ranks truth by +1.8e5 while PairAllSolved recovers it
    (+1.3). On clean data the two are equivalent, so robustness decides. This is
    what shipped as PyAutoLens#686.
  • Tensor source-plane weighting ranks truth first at both tiers; the scalar
    convention mis-maps the μ=367 image's radial noise.
  • Solved centres on both chi-squared flavours, with free centres documented
    as needing sampler muscle (the free-centre gradient plateau survives the
    phase-A logsumexp fix).
  • Two library defects found and fixed by the cluster gradient cells —
    PyAutoFit#1441 (traced attributes classified as pytree aux) and PyAutoLens#685
    (padded solver rows anchored at NaN-Jacobian centres). Both merged.

Evidence tail in this PR

The final commit closes the campaign out with the last three A100 jobs:

  • 331887 re-runs the cluster gradient cell that exposed both library
    defects. It now completes end-to-end (832 s, finite, no NaN), validating the
    merged fixes through a full search. But it still misses the basin
    (delta −1723.6 vs Nautilus +16.9), which closes the one question the notes had
    left open: the solved image-plane cell was the last candidate for a
    cluster-scale gradient-search recommendation, and it fails alongside the
    source-plane cells. At cluster scale, every objective tested is Nautilus's.
    Finite gradients were a correctness fix; they did not buy convergence.
  • 331888/331889 replace two cells that had been silent PyAutoFit resumes
    (sub-second sampler wall — identifiers ignore the dataset). Both reproduce
    their prior conclusions within scatter, so the resumes had not misled, but the
    shipped numbers now rest on searches that actually ran.
  • 331885/331886 are recorded as a deliberate non-result: both simple_extra
    arms timed out at 8h walls. A spurious position imposes an honest ~−1.4e4
    floor on every model, so Nautilus never converges. The DNF is symmetric,
    so this arm does not discriminate between the pairings — the default rests on
    the missing-image arm, which was already in hand.

Notes

  • The notes doc records two harness quirks that are easy to misread as defects
    of a given run: likelihood_evals is 65 for every MultiStartProdigy cluster
    cell (it counts vmapped batch calls), and best_fit is an AttributeError on
    every cluster cell including the Nautilus ones.
  • Results-only repo — no library or workspace code changes here. The workspace
    prose that cites these numbers is autolens_workspace#468.

🤖 Generated with Claude Code

Jammy2211 and others added 8 commits July 31, 2026 16:43
…discriminator cells, cluster searches, A100 submits

- Truth-anchored methodology: simulators write truth.json; every point_source/
  cluster search JSON now records results.truth_log_likelihood and
  delta_max_ll_vs_truth (evaluated with the cell's own fit class at the
  simulator truth), plus results.posterior_stats (per-param mean/std) for
  posterior-width comparisons.
- New model types: source_plane_tensor (free-centre weighting='jacobian',
  PyAutoLens#679) and image_plane_repeat_solved; six new galaxy leaf cells.
- Instrument variants for the pairing discriminator + domain-of-validity:
  simple_missing (image dropped), simple_extra (spurious position),
  near_caustic (source at 0.95x tangential caustic).
- Cluster tier searches: dataset_class='cluster' (factor-graph AnalysisPoint
  per system, cosmology pinned — solver custom_jvp constraint), ported model
  on the repo's dPIEMassB0Sph simulator; seven cluster leaf cells.
- FIX cluster simulator: solver.solve was called without plane_redshift, so
  every source's positions were solved to the FINAL plane — the z=1.0
  source's committed positions were unphysical (truth solved logL -4.2e6;
  +26.1 after the fix). Same bug exists in autolens_workspace
  cluster/simulator.py (flagged on autolens_workspace#436).
- 25 A100 submit scripts + submit_all_678_phase_b.sh driver (pre-generates
  datasets before sbatch to avoid the concurrent auto-simulate race).
…mpaign notes

Tensor ranks truth first at both tiers; free-centre plateau survives
logsumexp (solved confirmed); missing-image arm decisive for PairAll
(PairRepeatSolved mis-ranks truth by 1.8e5 logL); solved not overconfident
(exp-4); tensor holds at 0.95x caustic (exp-5). In flight: simple_extra 8h
reruns (331885/331886) + msp cluster image_plane_solved rerun (331887) on
the merged gradient fixes (PyAutoFit#1441, PyAutoLens#685).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnTmLoJjJgMTze5uAbg1Jd
…ean free-cell re-runs

Closes out the phase-B evidence campaign with the last three A100 jobs.

331887 (multi_start_prodigy/cluster/image_plane_solved): the cell that exposed
PyAutoFit#1441 and PyAutoLens#685 now completes end-to-end on the A100 — 832 s,
finite value, no NaN — confirming both merged fixes hold through a full search
and not just the 8-draw gradient probe. Its truth anchor (+14.658) matches the
Nautilus twin's exactly, cross-validating the run.

But the search still misses the basin (delta -1723.6 vs Nautilus +16.9). This
was the last candidate for a cluster-scale gradient-search recommendation, and
it fails alongside the source-plane cells (-14.8, -11062). The notes' caveat is
corrected accordingly: at cluster scale every objective tested is Nautilus's.
Making the gradients finite was a correctness fix; it did not buy convergence.
A negative delta is an optimizer defect, not a likelihood one, so none of this
touches the choice of default.

331888/331889 replace the two free `simple` Nautilus cells that had been silent
PyAutoFit resumes (sub-second sampler wall). Both reproduce their prior
conclusions within scatter — image-plane delta +2.27 -> +2.38, source-plane
+33476.5 -> +33474.5 against an identical truth anchor — so the resumes had not
misled, but the shipped numbers now rest on searches that actually ran. exp-4's
free einstein_radius std moves 0.0273 -> 0.0293 (solved bars ~28% wider, not
~40%; direction unchanged).

The two simple_extra exp-3 arms are recorded as a deliberate non-result: both
timed out at the 8h wall (331885/331886), f_live 0.92 and N_eff 12 on the
all-to-all arm. A spurious position imposes an honest ~-1.4e4 floor on every
model, so Nautilus cannot converge. The DNF is symmetric, so this arm does not
discriminate between the pairings — the missing-image arm is what the default
rests on, and it was already in hand.

Also records two harness quirks that are easy to misread as defects of this
run: likelihood_evals is 65 for every MultiStartProdigy cluster cell (it counts
vmapped batch calls), and best_fit is an AttributeError on every cluster cell
including the Nautilus ones.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The phase-B plane_redshift fix moved jitted_solve from a module-level function
into a per-iteration assignment from jitted_solve_for(), which made it a loop
variable that the jit_profile lambda captured by reference. Bound as a default
argument, matching the _tracer=tracer idiom already on the same line.

Not a live bug — jit_profile consumes the lambda inside the same iteration, so
the late binding never resolved to the wrong solve — but ruff B023 is right
that it is fragile, and this is what turned lint red on the branch. Verified
against main's unchanged file as a control: main passes B023, the branch failed,
so the branch introduced it rather than a ruff-version change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pure line-wrapping to satisfy the lint workflow's 'ruff format --check' step at
the repo's line-length = 100. No semantic change.

These are phase-B harness files, not the results commit. The branch had been
pushed since 2026-08-01 with no PR open, so neither ruff step had ever run
against it. Reproduced locally first — the same 3 files, byte-identical diffs —
so this is the repo's own formatter and not a CI-only ruff version difference.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ange

The range sits directly under a table whose first column is -33788.4, so it
reads as though all six source-plane cells fall in it. Five do; the galaxy
free-scalar cell is at +33474.5. Per the doc's own reading key that is the
LIKELIHOOD mis-ranking, not a search failure — Nautilus found its objective's
best basin fine, the basin is simply nowhere near truth. Stated explicitly so
the bias showcase is not misread as a Nautilus result.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cross-review of the open PRs (7 findings, all verified before acting).

MECHANISM WAS WRONG (F2). Both docs said repeat pairing 'has no way to leave a
model-predicted image unmatched'. It does — pair_repeat.py builds an
unmatched_model_mask and penalizes through unmatched_model_policy. Checked the
source rather than reasoning about it. The real mechanism: with an observed
image missing, the model image that would have paired to it is the nearest
neighbour of no observed position, so the default magnification_filter policy
sees a BRIGHT extra image (the exemption is for demagnified ones) and charges
its full distance to the nearest other observed position. Truth is punished for
correctly predicting an image the data lacks. Symptom and conclusion unchanged;
the explanation a reader would learn from was wrong.

UNSUPPORTED EVIDENCE (F3). The f_live=0.92 / N_eff=12 / logZ=-14238 figures for
the DNF arms are NOT reproducible from this PR. Went looking on RAL: the
surviving output trees hold only metadata, model.info, .identifier and a
12-line search.log with no sampler statistics, no Nautilus checkpoint, and no
SLURM stdout anywhere under /mnt/ral/jnightin. So the diagnostic is recorded
interpretation of a run whose output is gone, now labelled as such. What IS
reproducible is stated separately: sacct still reports both TIMEOUT, and the
clean siblings' JSONs give the 147.2 s / 162.7 s comparison walls. Also
concedes the epistemic point that a timeout proves non-completion within the
cap, not destroyed convergence.

UNATTRIBUTED PROBE (F4). '8/8 random cluster draws' has no artifact in this
repo; now points at the bug-fix PRs and the autofit_workspace_test#81
regression test as the citable form.

SELF-CONTRADICTION (F1). The note called solved+scalar a preference for wrong
models at delta +5.8, which its own reading key calls ordinary basin recovery.
Corrected to what the numbers show: the catastrophic scalar mis-ranking is a
FREE-CENTRE effect, solving the centre already removes most of it, and the
tensor is the further refinement that puts truth highest (+12.75 / +0.60 /
-33788).

WRONG WALL (F5). Nautilus cluster twin is 12.4 min total (742.08 s), not 12.6.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Jammy2211
Jammy2211 merged commit 272924a into main Aug 4, 2026
1 check passed
@Jammy2211
Jammy2211 deleted the feature/point-source-defaults-campaign branch August 4, 2026 20:59
@Jammy2211
Jammy2211 restored the feature/point-source-defaults-campaign branch August 6, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant