Skip to content

feat: Prodigy-vs-Nautilus point-source benchmark (#657 follow-up) - #99

Merged
Jammy2211 merged 1 commit into
mainfrom
feature/point-source-prodigy-benchmark
Jul 31, 2026
Merged

feat: Prodigy-vs-Nautilus point-source benchmark (#657 follow-up)#99
Jammy2211 merged 1 commit into
mainfrom
feature/point-source-prodigy-benchmark

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

Summary

Prodigy-vs-Nautilus benchmark on the point-source likelihoods (#657 follow-up, human-requested): two new free-centre cells mirroring the Nautilus anchors (pure sampler swap on the same 8-param PointFlux model/dataset), plus results for all six runs on the WSL dev laptop (CPU, viz disabled). Truth = simulator preset (er=1.6, centre (0,0)), truth-anchored logL computed per likelihood flavour.

cell max logL truth logL wall s best er / centre verdict
nautilus / source_plane −313.2 −33788 168 3.20 / (−0.37, −0.38) likelihood bias: wrong model beats truth by ~4 orders of magnitude in χ²
prodigy / source_plane −109.7 −33788 19.4 5.55 / (−0.63, −0.02) out-optimizes Nautilus 8.7× faster on the same (biased) objective
nautilus / image_plane +9.56 +7.20 739.7 1.623 / ~(0, 0) reference anchor — truth recovered
prodigy / image_plane (64×300) −79.9 +7.20 852.8 1.866 / (−0.02, 0.02) missed the 5 mas basin (start-count limited; -inf underflow plateaus)
prodigy / image_plane (256×300) −47.7 +7.20 3515.5 1.581 / (0.000, 0.001) truth basin found; final polish plateaued (best logL flat from step ~250)
prodigy / image_plane_solved (64×300) +2.37 +7.74 981.9 1.574 / (0.00, 0.01) truth recovered THROUGH the solver — the recommended config works

Conclusions

  1. The solver gradients demonstrably work in a real search: the solved-centre configuration (PointSolved + FitPositionsImagePairAllSolved, 5 params) converges to the truth basin with the default 64×300 budget, ~within 5 logL of the truth anchor, in ~16 min — gradients flowing through the lens-equation solve via the implicit-diff custom_jvp.
  2. The free-centre image-plane cell needs 4× the starts to find the same basin (and still under-polishes at 300 steps) — consistent with the filed PairAll -inf-underflow plateau issue shrinking the gradient-reachable basin, and with the solved variant's dimensionality advantage.
  3. The source-plane scalar-µ² likelihood is spectacularly biased on this dataset (truth logL −33788 vs −110 at wrong parameters): both samplers agree, so it's the objective, not the optimizer — direct evidence for the pairing guide's recommendation to search with the tensor-weighted solved fit and validate image-plane, and for the filed cluster PointSolved default swap.

Scripts Changed

  • scripts/point_source/searches/multi_start_prodigy/image_plane.py — new free-centre cell (mirrors nautilus/point_source/image_plane.py)
  • scripts/point_source/searches/multi_start_prodigy/source_plane.py — new free-centre cell (mirrors the nautilus anchor)
  • scripts/misc/searches/sweep.py — both registered in CELLS beside the anchors
  • results/searches/{nautilus,multi_start_prodigy}/point_source/** — the six result JSONs + PNGs above

Test Plan

  • ruff check clean; AUTOLENS_PROFILING_SMOKE=1 import-check green for both cells
  • All six cells executed end-to-end locally (this table)

Generated by the PyAutoLabs agent workflow.

…+ truth-anchored results (#657 follow-up)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Jammy2211 Jammy2211 added the pending-release Queued for the next release label Jul 31, 2026
@Jammy2211
Jammy2211 merged commit 4876054 into main Jul 31, 2026
1 check passed
@Jammy2211
Jammy2211 deleted the feature/point-source-prodigy-benchmark branch July 31, 2026 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release Queued for the next release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant