Skip to content

ideas.md

Latest commit

 

History

History
42 lines (34 loc) · 10.9 KB

ideas.md

File metadata and controls

42 lines (34 loc) · 10.9 KB
Raw

  • lens_calc_zero_contour_jax autolens workspace guide.

  • [from: ep-review-p1-F10] Adaptive/default EP damping: default delta < 1 (or DynamicUpdater as default) so undamped Laplace-only EP cannot silently collapse; consider a damping schedule that tightens as the KL step shrinks.

  • [from: ep-review-p5] Structured (block-Gaussian) mean field: allow correlated message blocks over variable groups instead of the fully-factorised q — directly addresses the exactness-vs-modularity trade-off documented for deterministic variables (z ⊥ parents under the current family).

  • [from: ep-review-p1] Power EP / alpha-divergence updates: the damped update is already an EMA on natural parameters; fractional (power-EP) updates are a small generalisation and handle heavy-tailed factors better than KL moment matching.

  • [from: ep-review-p4] EP health report: aggregate the new ep_history.csv into a per-factor/per-variable traffic-light summary (BAD_PROJECTION rates, oscillation detection, sigma-collapse flags) — the "single point of reference" view graphical_scoping.md limitation 3 asks for.

  • [from: ep-review-p3] Warm-start per-factor searches across EP cycles: each optimization_# currently restarts cold; seeding from the previous cycle's samples/posterior would cut the dominant per-cycle wall-time (ties to ep_scoping.md overhead findings).

  • [from: ep-review-p1] Audit + document StochasticEPOptimiser (expectation_propagation/stochastic.py, 117 lines) — out of the Phase 1 audit scope, undocumented, untested status unknown.

  • [from: ep-review-p1-F7] Evidence-correct EP model comparison: once the log_norm bookkeeping fixes land, validate EP log-evidence against nested-sampling evidence on toy graphs and expose it as a supported feature (currently unusable).

  • [from: ep-review-p6] Hierarchical exact updates: Gaussian parent–child HierarchicalFactor updates have closed forms; currently optimised numerically one factor per drawn variable (natural WP5 after PyAutoFit#1338's WP3).

  • [from: ep-review] Resumable EP: checkpoint/restore EPMeanField (+ EPHistory) so long multi-dataset fits survive session/HPC walltime boundaries — cosmology use case runs Nautilus-per-factor and would benefit first.

  • [from: ep-review] JAX-native Laplace factor fits: autodiff gradients/Hessians for the tilted distribution instead of quasi-Newton with numeric fallbacks — aligns EP with the wider JAX direction and removes the scipy-private line-search dependency (#1332 F9).

  • [from: ep-review-p3] HowToFit chapter 3 refresh: align tutorial 5 with the new step-by-step feature example, the diagnostics outputs (#1335), and swap TruncatedGaussianPrior usage once #1331-04/F6 fixes land (the flagship tutorial currently runs on the buggy truncated path).

  • [from: research jax-autodiff-gradients-audit (#87) · delaunay probe] Delaunay frozen-triangulation gradients: wrap _jax_delaunay_tables (PyAutoArray delaunay.py pure_callback) in jax.custom_jvp with a zero rule so value_and_grad stops hard-erroring, then FD-validate the frozen-triangulation gradients like the rectangular mesh; connectivity is piecewise-constant so the zero rule is the correct a.e. derivative. Coordinate with the nnls-solver-optimization claim on PyAutoArray.

  • [from: research jax-autodiff-gradients-audit (#87) · source_plane probe] Fix Grid2DIrregular.grid_2d_via_deflection_grid_from xp propagation so the point-source likelihood's forward jax.jit works (gradients already work; the jit gap blocks fast batched sampling).

  • [from: research jax-autodiff-gradients-audit (#87) · interferometer probe] Interferometer MGE FD correctness test: probe is 9/9 finite but no autodiff-vs-finite-difference validation exists for the visibility-space likelihood — add jax_grad/interferometer_mge.py on the pattern of the imaging suite.

  • [from: research jax-autodiff-gradients-audit (#87) · validated-likelihood set] NUTS/HMC sampler trial on the FD-validated likelihoods (weak lensing, point-source source-plane, imaging with RectangularUniform or parametric sources) via the sampler pipeline — gradients are now certified correct end-to-end, removing the main risk that stalled gradient-based inference.

  • [from: PyAutoBuild/to_do_list, evacuated 2026-07-10 by pyautoscientist-phase1] Test different numpy, scipy, scikit-learn, threadpoolctl, joblib versions.

  • [from: PyAutoBuild/to_do_list] Tests using full dynesty runs with inversion, BrightnessImageNN pix, with parallel processing.

  • [from: PyAutoBuild/to_do_list] Workspace tagging.

  • [from: PyAutoBuild/to_do_list] Remove annoying report.log and root.log once and for all.

  • [from: pyautoscientist-3b-smoke, 2026-07-10] Extend spawn --stamp-family to stamp the smoke thin-caller workflow + a starter smoke_install.sh into the template family (spec-gated on the reusable smoke workflow, which is now live).

  • [formalised -> feature/pyautogut/repo_skeleton.md] [from: research pyautogut-organ-decision · research/pyautobrain/pyautogut_organ_decision.md] Stand up the PyAutoGut repo skeleton — the new peer organ repo owning condemned self-material: the attic remote, the refs/archive/condemned/<name> archive-ref convention (stays out of git branch -a), and the condemned.md Mind manifest schema (type/locator/confidence/reason/merged?/condemned/sweep-after/breaks-if-wrong/archive-ref+SHA). Work-type feature/pyautogut; also adds PyAutoGut to repos.yaml + ORGANISM.md as a peer organ.

  • [formalised -> feature/pyautobrain/hygiene_pyautogut_drive_seam.md] [from: research pyautogut-organ-decision · research/pyautobrain/pyautogut_organ_decision.md] Add the hygiene→PyAutoGut drive seam in the Brain — a PyAutoGut-aware hygiene tidy mode that files condemned candidates into condemned.md async (no synchronous per-item gate, replacing the mid-sweep repo_cleanup interrogation), plus a batch sweep mode that runs the existing repo_cleanup safety gates against the manifest at a user-chosen time. Mirrors the Heart↔vitals template: the organ holds+voids, the conductor drives. Work-type feature/pyautobrain.

  • [formalised -> research/autofit/multi_band_factorgraph_compile_deeper_dig.md] [from: research multiband-compile · Experiment A · jax_compile/README.md multi-band section + project_multiband_factorgraph_compile_cliff memory] Multi-band FactorGraphModel value_and_grad compile — deeper dig + productize the fix. Experiment A (autolens_profiling branch research/multiband-compile-ab) proved the heterogeneous-per-band-shape cold compile is a 5.9× superlinear XLA-fusion cliff (homogeneous 4-band == single-band 120s; two distinct shapes = 704s CPU), that trace/eval are unchanged, and that the persistent cache rescues both arms (warm 2–7s). Follow-up, ONE task, profiling/research: (1) sub-investigation B — does a per-factor jit boundary inside PyAutoFit FactorGraphModel.log_likelihood_function bound cold to N×single-band + a linear combine, removing the superlinearity WITHOUT padding (the potential source lever, touches PyAutoFit); (2) sub-investigation C — add A100 multi-band vag rows (datacube_img / datacube_img_hetero) to the jax_compile census, quantify the far-lower absolute A100 cliff, and ship the final N-band verdict; (3) productize the immediate user workaround — pad short-wavelength bands to a common grid so all factors share one fused kernel — as a documented recipe and/or helper. Does NOT re-open the #71 single-band settings verdict. Repos: PyAutoFit, autolens_profiling. Difficulty large, supervised.

  • [from: research wsdev#117 pix-prodigy-cpu · pix_prodigy_findings.md] Warm-started sampler wave on meshes — revive the parked jax_native_posterior_sampler_wave with the MultiStartProdigy endpoint population (16 basin-labelled points) as initializer; mesh↔kernel mapping: ChEES-HMC/MCLMC on kernel-CDF+knn, tempered SMC (logZ) on delaunay.

  • [from: research wsdev#117 · resurrection diagnostics] Validate resurrection-rate → HMC-divergence-rate prediction per mesh — cheap; would give a principled mesh/kernel selection rule for Hamiltonian sampling on pixelized sources.

  • [from: research wsdev#117 · thin-baseline caveat] Heavier-settings Nautilus reference runs (n_live >= 500) on the knn/delaunay pix objectives — the n_live=100 baselines missed the truth basin; needed to claim "Prodigy matches a CONVERGED sampler" and to quantify the prior-volume search the gradient stage saves.

  • [from: research wsdev#117 · matern arm] Matérn kernel regularization as a production option — does its evidence ranking track AdaptSplit across real datasets? If yes the gradient-safe scheme can be the SLaM default, not just the search-time choice.

  • [from: research wsdev#117 · rect throughput anomaly] A100 profiling of the winning pix-Prodigy configs, starting with the rectangular kernel-CDF value_and_grad step-cost anomaly (~17x knn on CPU vs ~4.5x forward-eval ratio) and finishing the rect broad/bandwidth verdict GPU-side.

  • [from: research wsdev#117 · bandwidth discussion] Kernel-CDF bandwidth as annealing schedule vs free parameter — gate on a joint (bandwidth, reg) evidence scan at truth: monotone-to-zero ⇒ schedule ("search 1.0, refine 0.1", never a prior); interior optimum ⇒ bounded free param + FD-certify the bandwidth gradient first.

  • [from: #657 phase 3 validation] al.output_to_json serializes dict keys in nondeterministic order run-to-run (PYTHONHASHSEED-driven dict/set iteration somewhere in the autonerves/autoarray serialization path): regenerating a committed dataset JSON (e.g. autolens_workspace_test dataset/point_source/simple/*.json) produces value-identical but byte-different files, defeating byte-level determinism checks and adding VCS churn. Fix: sort keys (or preserve declared argument order) at the serializer, then regenerate committed JSONs once.

  • [from: #657 phase 4 ship] Maintainer-mode assistant wiki refresh once the solved variants are in a released version: update autolens_assistant/wiki/core/concepts/point_source.md + skills/al_point_source.md (currently a stub with no solved-variant coverage) via al_update_wiki — cover al.ps.PointSolved, the *Solved fit matrix, the tensor weighting conventions, and the cluster search-solved/validate-image-plane recommendation.

  • [from: #657 phase 5] Free cosmology parameters cannot cross the PointSolver implicit-diff custom_jvp boundary: Tracer registers cosmology as no_flatten aux, so a traced H0 arrives as a stale tracer (UnexpectedTracerError). Harmless for 2-plane positions fits (H0 does not move them); MULTI-PLANE gradient fits with free cosmology need cosmology flattened into the Tracer pytree (requires registered cosmology classes + auditing every jit(fit_from) user with a non-model cosmology instance).

  • Multi-core emcee/SneakyPool (number_of_cores=2) hangs on the WSL dev box while single-core is instant — pre-existing (reproduced on unchanged main autofit, py3.13, 2026-07-31 during the py314-fork-context smoke work; see complete/2026/07/py314-fork-context.md). Needs repro triage: real bug vs nproc=1 WSL quirk vs probe error.