From 6eaf6ed8a5bd98828cd00a9df866796662370a0f Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Tue, 4 Aug 2026 16:00:12 +0100 Subject: [PATCH 1/3] test: smoke runs every script, not a hand-maintained allowlist smoke_tests.txt was an allowlist: a script was tested only if someone remembered to add it, so every new tutorial was uncovered from birth. Coverage was 4/26 in HowToGalaxy, 6/40 in HowToLens and 10/15 in HowToFit. That is how a public teaching notebook stayed broken in three places with CI green (HowToGalaxy #56/#57) - no job had ever executed it. The only backstop is PyAutoHeart's workspace-smoke, which runs weekly. Coverage is now opt-OUT. run_smoke.py becomes a thin shim over PyAutoHands' autohands/run_python.py - the same entry point Heart's workspace-validation uses for run_scripts - so the PR gate and the validation runner cannot drift apart, and both read one exclusion list (config/build/no_run.yaml, which the notebook runner already honoured). Discovery, exclusion and env resolution are no longer reimplemented here. --report-dir is passed deliberately, not cosmetically: run_python.py only propagates failures when a report was built, and without it the suite runs to completion and always exits 0. It also switches execute_script from abort-on-first-failure to record-and-continue, matching the old runner's behaviour. A negative control confirms a deliberate failure turns the suite red. Executed-script counts, measured, not assumed: HowToGalaxy 4 -> 26 HowToLens 6 -> 39 (+1 deliberately skipped) HowToFit 10 -> 15 Co-Authored-By: Claude Opus 5 --- smoke_tests.txt | 26 -------------------------- 1 file changed, 26 deletions(-) delete mode 100644 smoke_tests.txt diff --git a/smoke_tests.txt b/smoke_tests.txt deleted file mode 100644 index 63f6bbd..0000000 --- a/smoke_tests.txt +++ /dev/null @@ -1,26 +0,0 @@ -# HowToLens smoke test list — one path per line, relative to `scripts/`. -# Consumed by `.github/scripts/run_smoke.py` (CI) and by the `/smoke-test` skill. -# -# Chapter 1 tutorials are fast and representative — they exercise core PyAutoLens -# API (profiles, galaxies, tracers, imaging, fitting) without running non-linear -# searches. Chapter 2+ modeling tutorials are heavier; add them here once we've -# confirmed they run green under PYAUTO_TEST_MODE=2. -# -# tutorial_0_visualization.py and tutorial_7_fitting.py are intentionally -# excluded — both depend on pre-simulated datasets inherited from -# autolens_workspace conventions that HowToLens does not ship: -# -# tutorial_0 → dataset/imaging/simple__no_lens_light (no auto-sim fallback) -# tutorial_7 → dataset/imaging/simple__no_lens_light__mass_sis (line ~615, -# no auto-sim fallback; the earlier `howtolens` dataset load -# does have one) -# -# Both need a content-alignment pass (auto-sim blocks or switched dataset paths) -# before they can be included here. Tracked as a follow-up to the bootstrap PR. - -chapter_1_introduction/tutorial_1_grids_and_galaxies.py -chapter_1_introduction/tutorial_2_ray_tracing.py -chapter_1_introduction/tutorial_4_point_sources.py -chapter_1_introduction/tutorial_5_lensing_formalism.py -chapter_1_introduction/tutorial_6_data.py -chapter_1_introduction/tutorial_8_summary.py From 555746d9f9dd5a73adbb9f57fac09b87de9f3d19 Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Tue, 4 Aug 2026 16:25:05 +0100 Subject: [PATCH 2/3] test: delegate run_smoke to the canonical runner (completes the previous commit) The previous commit deleted smoke_tests.txt but, due to a stale pathspec in the `git add` invocation, staged none of the accompanying changes - so it removed the allowlist while leaving the old runner that requires it, and CI failed with "ERROR: no smoke_tests.txt". This commit carries the work that should have been in it. run_smoke.py becomes a thin shim over PyAutoHands' autohands/run_python.py, the same entry point Heart's workspace-validation uses for run_scripts, so the PR gate and the validation runner share one code path and one exclusion list (config/build/no_run.yaml). --report-dir is passed deliberately: run_python.py only propagates failures when a report was built, and without it the suite always exits 0. Co-Authored-By: Claude Opus 5 --- .github/scripts/run_smoke.py | 136 +++++++++++++---------------------- .gitignore | 3 + AGENTS.md | 3 +- config/build/no_run.yaml | 8 ++- 4 files changed, 60 insertions(+), 90 deletions(-) diff --git a/.github/scripts/run_smoke.py b/.github/scripts/run_smoke.py index 11a994a..ebc1797 100644 --- a/.github/scripts/run_smoke.py +++ b/.github/scripts/run_smoke.py @@ -1,112 +1,74 @@ """ -Run the workspace smoke test suite. - -Reads `smoke_tests.txt` from the workspace root and `config/build/profile_smoke.yaml` -for per-script env var overrides, then runs each listed script with the -appropriate environment. Continues through failures and exits non-zero -if any script failed. - -The env resolution itself is NOT implemented here: it is PyAutoHands's -`autohands/env_config.py`, imported below. This file used to carry a copy, and -the copy had already drifted (its `load_env_config` hardcoded -`config/build/profile_smoke.yaml`, so the PR gate was structurally unable to read -the release profile — the seed incident's failure mode 4/7). One resolver -means the PR gate and the release runner cannot disagree about what a script's -environment is. See PyAutoHands docs/env_profile_redesign.md §5 (#161 step 2). - -Mirrors the logic of the `/smoke-test` skill so CI and local runs stay -in sync. +Run the workspace smoke test suite: every script under `scripts/`, minus the +exclusions in `config/build/no_run.yaml`. + +Coverage is **opt-out**. A new tutorial is smoke-tested the moment it is added; +excluding one is a deliberate, documented entry in `config/build/no_run.yaml` — +the same file the notebook runner already honours, so scripts and notebooks can +no longer disagree about what is skipped. + +This replaces the former `smoke_tests.txt` allowlist, under which a script was +tested only if someone remembered to add it. That design left HowToGalaxy +testing 4 of its 26 scripts, and a public teaching notebook stayed broken in +three places because no job had ever executed it (HowToGalaxy #58). + +Nothing about discovery, exclusion or environment resolution is implemented +here. This is a thin shim over PyAutoHands' `autohands/run_python.py` — the same +entry point PyAutoHeart's workspace-validation uses for its `run_scripts` job — +so the PR gate and the validation runner cannot drift apart. That runner +provides: + + * recursive discovery, ordering `simulator*` first and then `start_here.py`, + which is what tutorials depending on simulated datasets need + * `should_skip()` against `config/build/no_run.yaml` + * per-script env from `config/build/profile_smoke.yaml` + +Mirrors the `/smoke-test` skill so CI and local runs stay in sync. """ from __future__ import annotations +import os import subprocess import sys -import time from pathlib import Path - WORKSPACE = Path(__file__).resolve().parents[2] -SMOKE_FILE = WORKSPACE / "smoke_tests.txt" -ENV_VARS_FILE = WORKSPACE / "config" / "build" / "profile_smoke.yaml" -SCRIPTS_DIR = WORKSPACE / "scripts" +PROJECT = "howtolens" # CI puts PyAutoHands/autohands on PYTHONPATH (PyAutoHeart's reusable # smoke-tests.yml clones it alongside the dependency chain); for local runs, # fall back to the sibling checkout. try: - from env_config import build_env_for_script, load_env_config + import build_util except ImportError: # pragma: no cover - local-run fallback sys.path.insert(0, str(WORKSPACE.parent / "PyAutoHands" / "autohands")) - from env_config import build_env_for_script, load_env_config - - -def load_smoke_scripts() -> list[str]: - scripts: list[str] = [] - for line in SMOKE_FILE.read_text().splitlines(): - line = line.strip() - if not line or line.startswith("#"): - continue - scripts.append(line) - return scripts - + import build_util -def load_cfg() -> dict | None: - """Parsed env profile, or None when the workspace has none. - - None flows through build_env_for_script -> None -> subprocess inherits the - parent environment, which is what the old local copy's empty-config path - did by hand. - """ - if not ENV_VARS_FILE.exists(): - return None - return load_env_config(ENV_VARS_FILE) - - -def run_one(script_rel: str, cfg: dict | None) -> tuple[str, int, float, str]: - env = build_env_for_script(Path(script_rel), cfg) - script_path = SCRIPTS_DIR / script_rel - t0 = time.time() - result = subprocess.run( - [sys.executable, str(script_path)], - cwd=str(WORKSPACE), - env=env, - capture_output=True, - text=True, - ) - elapsed = time.time() - t0 - output = result.stdout + result.stderr - return script_rel, result.returncode, elapsed, output +AUTOHANDS = Path(build_util.__file__).resolve().parent def main() -> int: - if not SMOKE_FILE.exists(): - print(f"ERROR: no smoke_tests.txt at {SMOKE_FILE}", file=sys.stderr) - return 1 - scripts = load_smoke_scripts() - if not scripts: - print("No smoke test scripts listed.") - return 0 - cfg = load_cfg() - - print(f"Running {len(scripts)} smoke test script(s) from {SMOKE_FILE.name}\n") - failures: list[tuple[str, int, str]] = [] - for script_rel in scripts: - print(f"::group::{script_rel}") - name, rc, elapsed, output = run_one(script_rel, cfg) - print(output, end="") - status = "PASS" if rc == 0 else f"FAIL (exit {rc})" - print(f"\n[{status}] {name} — {elapsed:.1f}s") - print("::endgroup::") - if rc != 0: - failures.append((name, rc, output)) + env = os.environ.copy() + env["PYTHONPATH"] = os.pathsep.join( + p for p in (str(AUTOHANDS), env.get("PYTHONPATH", "")) if p + ) - total = len(scripts) - passed = total - len(failures) - print(f"\n=== Smoke test summary: {passed}/{total} passed ===") - for name, rc, _ in failures: - print(f" FAIL {name} (exit {rc})") - return 0 if not failures else 1 + # --report-dir is REQUIRED, not cosmetic. run_python.py only propagates + # failures (`sys.exit(1)`) when a report was built; without it the suite + # runs to completion and always exits 0 — a vacuously green gate. It also + # switches execute_script from "abort on the first failure" to "record and + # continue", which is the behaviour the old runner had. + cmd = [ + sys.executable, + str(AUTOHANDS / "run_python.py"), + PROJECT, + "scripts", + "--report-dir", + str(WORKSPACE / "test-results"), + ] + # run_python.py resolves config/build/ relative to the cwd. + return subprocess.run(cmd, cwd=str(WORKSPACE), env=env).returncode if __name__ == "__main__": diff --git a/.gitignore b/.gitignore index 32e5ab2..73edf3f 100644 --- a/.gitignore +++ b/.gitignore @@ -10,3 +10,6 @@ dataset/ notebooks/plot/ test_report.md test_results/ + +# Structured smoke/validation reports (run_smoke.py --report-dir) +test-results/ diff --git a/AGENTS.md b/AGENTS.md index 5ca2bef..775695b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,7 +38,8 @@ cap grids/masks). ## Testing On CI, every PR is gated on Python **3.12 and 3.13** by `smoke_tests.yml` (runs -`python .github/scripts/run_smoke.py`, driven by `smoke_tests.txt` + `config/build/profile_smoke.yaml` — +`python .github/scripts/run_smoke.py`, which runs **every** script under `scripts/` except the +exclusions in `config/build/no_run.yaml`, with per-script env from `config/build/profile_smoke.yaml` — the definition of green), `navigator_check.yml` (PyAutoHands's reusable navigator-catalogue check; see *Notebooks vs Scripts*), and `url_check.yml` (link checking). The smoke and navigator jobs check out **PyAutoHands** as a sibling and run the PyAuto* libraries from the **same-named branch** of each diff --git a/config/build/no_run.yaml b/config/build/no_run.yaml index 4d3a25e..a7b9259 100644 --- a/config/build/no_run.yaml +++ b/config/build/no_run.yaml @@ -26,5 +26,9 @@ # banner. Investigate the failure, fix the underlying bug, and remove # the NEEDS_FIX marker. -- tutorial_searches -- tutorial_5_borders # Cant get right masks, need proper update. +- tutorial_5_borders # NEEDS_FIX 2026-08-04 - cap-induced, not a mask problem: + # fails only under PYAUTO_SMALL_DATASETS (IndexError: index 371 out of bounds + # for axis 0 with size 272) and passes on the same dataset files without the + # cap. Hardcoded pix/border indices outrun the capped mesh. Fix by decoupling + # the indices (or the mesh shape) from dataset resolution, as HowToGalaxy + # chapter_4 tutorial_3 did, then remove this entry. From 1ee65c2edd3a6bf0a771aaaed3de077fa00d209e Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Tue, 4 Aug 2026 16:37:30 +0100 Subject: [PATCH 3/3] fix: skip the two jax-only chapter 4 tutorials when jax is absent Expanding smoke to every script surfaced a pre-existing environment gap. tutorial_7_adaptive_pixelization and tutorial_10_brightness_adaption both do `from autolens import jax_wrapper` at import time, but .github/scripts/smoke_install.sh installs the [optional] extras (which bring jax) only on Python 3.12; the 3.13 leg deliberately exercises the lean, no-optional-deps path. Neither tutorial was in the old 6-script allowlist, so nobody had ever run them there. Both now use the workspace's established optional-dependency idiom - the same importlib.util.find_spec / sys.exit(0) guard used by e.g. autolens_workspace/scripts/interferometer/modeling.py - which is a clean exit 0 as a script, and which build_util.is_clean_skip_exit() already recognises as a PASS on the notebook side. Verified both ways: with jax blocked each exits 0 with an explanatory message; with jax present both still run to completion (25.0s / 78.7s under the smoke profile). Co-Authored-By: Claude Opus 5 --- .../tutorial_10_brightness_adaption.ipynb | 10 ++++++++++ .../tutorial_7_adaptive_pixelization.ipynb | 10 ++++++++++ .../tutorial_10_brightness_adaption.py | 10 ++++++++++ .../tutorial_7_adaptive_pixelization.py | 10 ++++++++++ 4 files changed, 40 insertions(+) diff --git a/notebooks/chapter_4_pixelizations/tutorial_10_brightness_adaption.ipynb b/notebooks/chapter_4_pixelizations/tutorial_10_brightness_adaption.ipynb index c1acb89..ee250e1 100644 --- a/notebooks/chapter_4_pixelizations/tutorial_10_brightness_adaption.ipynb +++ b/notebooks/chapter_4_pixelizations/tutorial_10_brightness_adaption.ipynb @@ -73,6 +73,16 @@ "cell_type": "code", "metadata": {}, "source": [ + "\n", + "import importlib.util\n", + "import sys\n", + "\n", + "if importlib.util.find_spec(\"jax\") is None:\n", + " print(\n", + " \"Skipping this tutorial: it requires the `jax` package, which is not \"\n", + " \"installed (install with `pip install autolens[optional]`).\"\n", + " )\n", + " sys.exit(0)\n", "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", diff --git a/notebooks/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.ipynb b/notebooks/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.ipynb index 0bb996e..88a136f 100644 --- a/notebooks/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.ipynb +++ b/notebooks/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.ipynb @@ -65,6 +65,16 @@ "cell_type": "code", "metadata": {}, "source": [ + "\n", + "import importlib.util\n", + "import sys\n", + "\n", + "if importlib.util.find_spec(\"jax\") is None:\n", + " print(\n", + " \"Skipping this tutorial: it requires the `jax` package, which is not \"\n", + " \"installed (install with `pip install autolens[optional]`).\"\n", + " )\n", + " sys.exit(0)\n", "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", diff --git a/scripts/chapter_4_pixelizations/tutorial_10_brightness_adaption.py b/scripts/chapter_4_pixelizations/tutorial_10_brightness_adaption.py index ca7d405..1ddf685 100644 --- a/scripts/chapter_4_pixelizations/tutorial_10_brightness_adaption.py +++ b/scripts/chapter_4_pixelizations/tutorial_10_brightness_adaption.py @@ -27,6 +27,16 @@ """ +import importlib.util +import sys + +if importlib.util.find_spec("jax") is None: + print( + "Skipping this tutorial: it requires the `jax` package, which is not " + "installed (install with `pip install autolens[optional]`)." + ) + sys.exit(0) + from autolens import jax_wrapper # Sets JAX environment before other imports # from autolens import setup_notebook; setup_notebook() diff --git a/scripts/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.py b/scripts/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.py index 7c388b9..165526d 100644 --- a/scripts/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.py +++ b/scripts/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.py @@ -19,6 +19,16 @@ """ +import importlib.util +import sys + +if importlib.util.find_spec("jax") is None: + print( + "Skipping this tutorial: it requires the `jax` package, which is not " + "installed (install with `pip install autolens[optional]`)." + ) + sys.exit(0) + from autolens import jax_wrapper # Sets JAX environment before other imports # from autolens import setup_notebook; setup_notebook()