diff --git a/AGENTS.md b/AGENTS.md index 60e6115..6ec92f1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -103,7 +103,7 @@ reference, with its own `[[wiki-link]]` schema — arrives in a later phase (`PE it does, do not cite it and do not invent its contents. "The wiki" means `wiki/core/` unless `project/` is named. `wiki/core/` now has `stack/`, `api/`, `concepts/`, `operations/` and `external/`; [`wiki/core/index.md`](./wiki/core/index.md) lists every page that exists and -states plainly what is still missing (the dataset-layout and HPC operations pages). +states plainly what is still missing (the two HPC operations pages). --- @@ -276,13 +276,18 @@ Load operational references on demand, not every session: - **The reference wiki itself** → [`wiki/core/index.md`](./wiki/core/index.md) lists every page that exists, by section. -Two operational references are still unwritten (`PENDING.md` lists both with their grounding -scripts). Until they land, use the ground truth directly rather than citing a page that does -not exist: +- **Dataset layout + `info.json`** → + [`wiki/core/operations/dataset.md`](./wiki/core/operations/dataset.md) — the + `wavebands//` convention, the `info.json` schema, loading one waveband, and the + bundled dataset's sky and PSF caveats. **One dataset ships with this repo**: + `dataset/imaging/cosj100020+015344/`, a four-band real JWST/NIRCam cutout of an early-type + galaxy at z = 0.3422 — its own `README.md` there is the authority for its provenance, and + the real-data gate above applies to it in full. + +The two HPC operational references are still unwritten (`PENDING.md` lists both with their +grounding scripts). Until they land, use the ground truth directly rather than citing a page +that does not exist: -- **Dataset layout + `info.json`** → the `dataset/imaging//` trees and per-dataset - READMEs in `autogalaxy_workspace`, plus - `autogalaxy_workspace:scripts/imaging/data_preparation/start_here.py`. - **HPC science** (cores, JAX/GPU, SLURM concepts) → `autogalaxy_workspace:scripts/guides/hpc/example_cpu_and_gpu.py` and `autogalaxy_workspace:scripts/guides/using_jax.py`. The `hpc/` infrastructure folder is not diff --git a/PENDING.md b/PENDING.md index 3257b76..fba407d 100644 --- a/PENDING.md +++ b/PENDING.md @@ -26,25 +26,34 @@ named. ## Phase 2 — dataset, README front door, external signposts -- [ ] `dataset/imaging//` — a real multi-band JWST COSMOS-Web NIRCam cutout of a - non-lens galaxy: `wavebands//{data,noise_map,psf}.fits` + `info.json`, plus a - per-dataset provenance README. Every `info.json` number measured by a committed script - or cited — never invented. Gated on the user approving the cutout + archive identifiers. -- [ ] `dataset/imaging//mask_extra_galaxies.fits` — only if the cutout has real - neighbours. Grounding: `imaging/data_preparation/gui/mask_extra_galaxies.py`. -- [ ] `wiki/core/operations/dataset.md` — on-disk dataset layout and `info.json`. **Moved here - from Phase 3**: the page describes the bundled dataset's tree, so it cannot be written - honestly before that dataset exists. Grounding: - `imaging/data_preparation/start_here.py` and the workspace `dataset/` trees. Until it - lands, `wiki/core/index.md` names it as pending and routes to those two sources. -- [ ] `docs/make_readme_figures.py` + the hero PNG it renders + a vendored-sources README. - `docs/` is outside the audit scan set, so it is validated by execution from a fresh venv. -- [ ] `README.md` v2 — science framing, three anchored example prompts, and an explicit "what - works today / what's coming" section. -- [ ] `benchmarks/prompts/easy__sersic.md` — the easy assistant card, whose - prompt text must match the README verbatim (enforced by - `autoassistant/tests/test_benchmark.py::test_repo_readme_prompts_match_cards`, currently - skipped). Grounding: `imaging/start_here.py`. +**Delivered.** `dataset/imaging/cosj100020+015344/` ships a real four-band JWST/NIRCam cutout +reduced through `PyAutoReduce` from COSMOS-Web (program 1727) exposures: `F115W` and `F150W` at +0.03"/pixel, `F277W` and `F444W` at 0.06"/pixel, each with `{data,noise_map,psf}.fits` and its +own `info.json`, plus a dataset-level `info.json`, a `reduction_manifest.json` and a +full-provenance `README.md`. Every `info.json` number is measured from the delivered cutouts; +the redshift and catalogue identifiers are cited rather than measured, and the sky pedestal and +model-PSF caveats are stated rather than smoothed over. +`wiki/core/operations/dataset.md` documents the layout, the schema and those caveats, and is +listed in `wiki/core/index.md`. `docs/make_readme_figures.py` rebuilds the README hero +(`docs/images/cosj100020+015344_dataset.png`) offline from the shipped FITS alone — no +third-party asset is vendored, which `docs/images/sources/README.md` records explicitly. +`README.md` v2 is the real front door, and `benchmarks/prompts/easy_cosj100020_imaging.md` +freezes its second starter prompt with a 100-point rubric — which un-skips the two repo-level +tests in `autoassistant/tests/test_benchmark.py`. + +The card shipped as `easy_cosj100020_imaging.md`, not the planned +`easy__sersic.md`: its prompt fits an MGE bulge with a free sky level and adds +a single Sersic only as a comparison, so naming it for one profile would have described less +than it measures. + +### Still open + +- [ ] `dataset/imaging/cosj100020+015344/mask_extra_galaxies.fits` — the bundled cutout *does* + have a real neighbour, so this line stays live: a faint source **2.6" from the centre**, + inside any mask wide enough to reach the galaxy's outer isophotes. (A brighter one 8.0" + out is already excluded by a <~4" mask.) No mask ships today, so that source has to be + masked or modelled per session — the dataset README and the README hero figure both flag + it. Grounding: `imaging/data_preparation/gui/mask_extra_galaxies.py`. - [ ] External signpost PRs (shipped from their own repos, merged after this phase): `autogalaxy_workspace` llms.txt + README, `PyAutoGalaxy/llms.txt`, `HowToGalaxy/llms.txt`, and the org profile README — four places that currently advertise this assistant. diff --git a/README.md b/README.md index 323af2f..5fe5c26 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,237 @@ # PyAutoGalaxy Assistant -An AI assistant that lets you use natural language to do galaxy structure -modeling with [PyAutoGalaxy](https://github.com/PyAutoLabs/PyAutoGalaxy) — -fitting the light of galaxies in imaging and interferometer data: light -profiles, multi-Gaussian expansions, bulge–disk decomposition, ellipse / -isophote fitting, pixelized reconstructions of clumpy galaxies, and -multi-wavelength analysis. - -**This assistant is being built now.** The phased build is tracked at -[PyAutoBrain#188](https://github.com/PyAutoLabs/PyAutoBrain/issues/188); -skills, curated API reference, science wiki, bundled datasets and onboarding -prompts land over the coming phases. - -Until it is complete, the mature sibling assistant for strong gravitational -lensing — [autolens_assistant](https://github.com/PyAutoLabs/autolens_assistant) -— shows what this repo will become, and the -[autogalaxy_workspace](https://github.com/PyAutoLabs/autogalaxy_workspace) -(start from its `llms.txt`) is the place to run PyAutoGalaxy examples today. +This repository is the **PyAutoGalaxy Assistant**: an AI assistant which **lets you use +natural language** to do galaxy structure and morphology science with +[PyAutoGalaxy](https://github.com/PyAutoLabs/PyAutoGalaxy) — fitting the light of galaxies +in imaging and interferometer data. Surface-brightness profiles, multi-Gaussian expansions, +bulge–disk decomposition, isophote and ellipse fitting, pixelised reconstructions of clumpy +galaxies, and multi-wavelength analysis. + +## Getting Started + +To illustrate the `autogalaxy_assistant` we will use James Webb Space Telescope NIRCam +imaging of **COSJ100020+015344**, a galaxy whose four-band cutout ships with this +repository in `dataset/imaging/cosj100020+015344`: + + + +This is a **single bright, smooth, early-type galaxy** at a spectroscopic redshift of +z = 0.3422 — one object, four wavebands, no complications from a companion of comparable +brightness. That makes it a clean first target, and it also makes the three things you *do* +have to decide impossible to hide behind: + +- **How far out does your mask reach?** A mask that truncates the outer isophotes biases the + effective radius and Sersic index directly, so the 4" circle drawn above is an opening + suggestion rather than an answer. Your first interactions with the assistant will ask you + about it. +- **The sky has not been subtracted.** The delivered data carry the real JWST background as a + positive pedestal — 5 to 19 times the noise, depending on the band. A light-profile fit + that ignores it absorbs the pedestal into the profile wings and hands back an inflated size + and Sersic index. Free `background_sky_level` on an `af.Model(ag.DatasetModel)`, or subtract + the measured value first. +- **There is a faint neighbour 2.6" from the centre**, inside any mask wide enough to reach + the galaxy's outer light. It has to be masked or modelled, not ignored. + +The galaxy's own measured shape — axis ratio ~0.83–0.89 and position angle ~90–99°, drifting +with wavelength — is in the zoom row above, and every number in the figure is read straight +from the dataset's `info.json`. The full provenance, from MAST exposures through the +reduction to each measurement, is in +[the dataset's README](dataset/imaging/cosj100020+015344/README.md) — **including the PSF +caveat that matters most**: the shipped kernel is a model (STPSF) PSF rather than an +empirical one, and it is the dominant systematic in any fit to this data. + +There are two ways to use `autogalaxy_assistant`, choose whichever best suits how you work +with AI. + +### AI Chat Assistant + +Ask questions to a conversational AI assistant such as **ChatGPT** or **Claude** in a +desktop browser. + +This requires two things: + +- Make sure your assistant has a **GitHub connector** enabled so it can read this + repository, and give it this repository's URL + (https://github.com/PyAutoLabs/autogalaxy_assistant) in your opening prompt. +- Point the assistant explicitly at [`llms.txt`](llms.txt), which gives it the start-up + instructions for how `autogalaxy_assistant` works. Connectors do not reliably fetch that + file on their own, and results are markedly better when it is named. + +So prefix either starter prompt below with: + +``` +Use the autogalaxy_assistant (www.github.com/PyAutoLabs/autogalaxy_assistant) with the +GitHub connector, first reading its llms.txt file for initial start up. +``` + +A chat assistant cannot run code or inspect the `.fits` files on your machine, so it will +plan the analysis, explain the physics and draft the scripts — and it will ask *you* to plot +and confirm the data before it composes a fit. Running the fit is where a coding agent takes +over. + +### AI Coding Agent (CLI) + +`autogalaxy_assistant` has first-class support for AI coding agents such as **Claude Code** +and **Codex**. + +A coding agent is a command-line tool that runs locally in your terminal. It can inspect +your `.fits` data, write and execute Python, perform end-to-end galaxy modelling, and load +existing results from your computer for inspection. + +To start, clone the repository: + +```bash +git clone https://github.com/PyAutoLabs/autogalaxy_assistant.git +cd autogalaxy_assistant +``` + +Then open your AI coding agent in your terminal inside the `autogalaxy_assistant` folder you +just cloned. If `PyAutoGalaxy` is not already installed, the coding agent will use +`autogalaxy_assistant` to install it after you submit your first prompt. + +## Two Starter Prompts + +Both are grounded on the bundled dataset, so they work immediately after cloning — copy, +paste, and go. + +### Starter Prompt 1 — new to PyAutoGalaxy + +Plot the data, then get the lie of the land before committing to a model. This is the prompt +to use if you have not fitted a surface-brightness profile before; the assistant will lead +with the physics and point you at the tutorial series. + +``` +Find the bundled JWST imaging of the galaxy COSJ100020+015344 in +dataset/imaging/cosj100020+015344, give me a short script that plots all four wavebands, +and then — since I'm new to PyAutoGalaxy — give me an overview of the different ways we +could model the structure of an early-type galaxy at z = 0.34. +``` + +### Starter Prompt 2 — experienced user, end-to-end fit + +A real multi-band structural measurement, including the sky pedestal this data actually +has. Expect the assistant to ask you about the mask extent and the faint neighbour before +it starts fitting — that gate is deliberate and is not waived on any harness. + +``` +Assistant mode. + +Fit the JWST F277W imaging in dataset/imaging/cosj100020+015344 with a multi-Gaussian +expansion (MGE) bulge, freeing the background sky level — the sky is not subtracted in +this data. Report the effective radius, axis ratio and position angle, add a single-Sersic +fit so I get a Sersic index to compare against, and then tell me how the recovered +structure changes across the other three wavebands. +``` + +This prompt is also shipped as a frozen benchmark card, +[`benchmarks/prompts/easy_cosj100020_imaging.md`](benchmarks/prompts/easy_cosj100020_imaging.md), +with a scoring rubric — the two texts are kept identical by a unit test. + +## Customize Your Assistant + +`autogalaxy_assistant` adapts its behaviour to suit your prompt, whether you are using a +conversational assistant (e.g. ChatGPT) or a coding agent (e.g. Claude Code): + +- Want to plan your analysis and compare the available approaches before running anything? + Simply say so in your opening prompt. + +- Want the assistant to ask questions as it goes, helping you understand the analysis and + make informed choices? Ask it to guide you through the process. + +- Want it to complete a task end-to-end without consulting you? Tell it to **one-shot** the + task. + +If you are new to galaxy structure and morphology — particularly an undergraduate or +early-stage PhD student — ask the assistant to use **Teacher Mode**. It will explain the +fundamentals of surface-brightness fitting in greater detail, and link you to relevant, +human-readable documentation so you can see what **PyAutoGalaxy** is doing. + +## What Works Today, and What Is Coming + +This assistant is **public from birth and built in phases**, so this section is a status +report rather than a feature list. [`PENDING.md`](PENDING.md) is the authoritative ledger of +everything still missing, with the `autogalaxy_workspace` script that will ground each item. +No file in this repository links to something that does not exist yet. + +**Live today:** + +- **The core modelling loop — nine skills.** Environment setup, imaging data preparation, + dataset simulation, model building, search configuration, running the fit, plotting the + fit, loading results, and debugging a failed fit. A galaxy-science request routes into + these. Alongside them sit two meta-skills, two project-workflow skills and three + repository-maintenance skills — sixteen in total, catalogued in + [`skills/README.md`](skills/README.md). +- **The curated reference wiki**, [`wiki/core/`](wiki/core/index.md) — 37 pages across + `stack/`, `api/`, `concepts/`, `operations/` and `external/`. Every page pins the source + commits it was validated against. +- **This bundled dataset** — four real JWST NIRCam wavebands with full provenance, plus + [`wiki/core/operations/dataset.md`](wiki/core/operations/dataset.md) documenting the + on-disk layout and the `info.json` schema. +- **One benchmark card** — Starter Prompt 2 above, frozen with a 100-point rubric under + [`benchmarks/`](benchmarks/README.md). + +**Still being written:** + +- **Feature skills beyond a single smooth profile** (Phase 4b): basis profiles and MGE, + pixelisations, ellipse fitting, multi-dataset and interferometer modelling, multi-galaxy + and cluster fields, and search chaining. The underlying PyAutoGalaxy features all exist and + the reference wiki covers them; what is pending is the dedicated procedural skill for each. +- **The literature wiki** (Phase 5) — a galaxy-structure science reference with cross-linked + concept, entity and source pages. Until it lands, the assistant will not cite it and must + not invent its contents. +- **The remaining benchmark cards, and HPC support** (Phase 6) — medium, hard and teacher + cards, plus the `hpc/` templates and sync CLI for cluster runs. + +The phased build is tracked at +[PyAutoBrain#188](https://github.com/PyAutoLabs/PyAutoBrain/issues/188). + +### Supported Coding Agents + +CLI coding agents like Claude Code and Codex may require a paid subscription. The table +below shows the agents `autogalaxy_assistant` has been tested with and whether they offer a +free plan — though pretty much any coding agent should work. + +| Interface | Support | Access and cost | Notes | +|---|---|---|---| +| **Claude Code** | Primary; thoroughly tested | Normally a [paid Claude subscription or metered API usage](https://code.claude.com/docs/en/costs). | Loads the canonical instructions through `CLAUDE.md`. | +| **Codex CLI** | Primary; thoroughly tested | A [limited free plan](https://developers.openai.com/codex/pricing/) may be available; paid plans or API billing provide more usage. | Reads `AGENTS.md` directly and can edit and run the project locally. | +| **Gemini CLI** | Supported | Offers [limited free quotas](https://github.com/google-gemini/gemini-cli/blob/main/docs/resources/quota-and-pricing.md); subscriptions or usage billing provide higher limits. | Loads the repository instructions through `.gemini/settings.json`. | +| **OpenCode** | Supported | The client is open source; model-provider access may be free or paid. | Use it from the repository root so it can discover the project context. | + +## Science Project + +When you begin a specific scientific study, `autogalaxy_assistant` can create a dedicated +**science project**: a separate, logically structured repository holding that study's +datasets, configuration, analysis scripts, results, plotting scripts and the transcript of +your work with the assistant. Every generated script is documented well enough to convert +automatically into a Jupyter notebook, its explanations becoming markdown cells. Linking it +to GitHub gives collaborators a straightforward way to inspect the project's state and build +on it, and if the study leads to a paper the repository can serve as its open-source +companion. + +The assistant itself stays the copilot; the project is its own repo. To start one, just say +so: + +``` +Start a science project for my COSJ100020+015344 structural analysis. +``` + +The workflow is owned by the +[`start-new-project`](skills/start-new-project.md) skill. + +## The Sibling Assistant + +If your science is strong gravitational lensing rather than galaxy structure, the mature +sibling assistant is +[autolens_assistant](https://github.com/PyAutoLabs/autolens_assistant) — same architecture, +built on PyAutoLens. The two are independent; use whichever matches your science. ## License -MIT — see [LICENSE](LICENSE). +This repository is released under the [MIT License](LICENSE), consistent with the wider +PyAuto\* ecosystem. The assistant ships agent instructions and reference material derived +from the public PyAuto\* repositories; the underlying libraries are released under their own +licenses (see each repo). The bundled dataset was reduced from public JWST archival +observations — see +[its README](dataset/imaging/cosj100020+015344/README.md) for the archive identifiers and +catalogue citations. diff --git a/benchmarks/README.md b/benchmarks/README.md index cdb0344..f4ebe02 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -23,15 +23,14 @@ benchmarks/ ## The benchmarks -**No cards are written yet.** `benchmarks/prompts/` is empty: the easy card is authored in -Phase 2 alongside the bundled dataset and the README prompts it must mirror, the rest in -Phase 6. The repo-root `PENDING.md` tracks them. The planned set below is a plan, so the -entries are deliberately unlinked, and the two repo-level tests in -`autoassistant/tests/test_benchmark.py` skip until the first card lands. +**One card is written.** The easy card landed in Phase 2 alongside the bundled dataset and the +README prompt it mirrors; the remaining three are authored in Phase 6 and the repo-root +`PENDING.md` tracks them. Planned entries below are deliberately unlinked because their files +do not exist yet. | Card | Mode | Difficulty | Exercises | |------|------|-----------|-----------| -| `prompts/easy__sersic.md` _(planned, Phase 2 — named with the dataset)_ | assistant | easy | the built-in workflow on the bundled JWST imaging: inspect, mask, fit a single Sersic, read the result | +| [`prompts/easy_cosj100020_imaging.md`](prompts/easy_cosj100020_imaging.md) | assistant | easy | the built-in workflow on the bundled JWST imaging: inspect, mask, fit an MGE bulge with a free sky level, add a Sersic for the index, compare wavebands | | `prompts/medium_mge_bulge_disk.md` _(planned, Phase 6)_ | assistant | medium | pipeline design beyond bundled workflows: Bayesian model comparison of an MGE against a bulge+disk decomposition, runtime/HPC judgment | | `prompts/hard_multi_galaxy_cluster.md` _(planned, Phase 6)_ | assistant | hard | cross-package synthesis: multi_galaxy × cluster × imaging × interferometer, simulation + joint modeling | | `prompts/teacher_workflow.md` _(planned, Phase 6)_ | teacher | easy | pedagogy: end-to-end workflow walkthrough on simulated data | diff --git a/benchmarks/prompts/easy_cosj100020_imaging.md b/benchmarks/prompts/easy_cosj100020_imaging.md new file mode 100644 index 0000000..b3cfdc9 --- /dev/null +++ b/benchmarks/prompts/easy_cosj100020_imaging.md @@ -0,0 +1,95 @@ +--- +id: assistant-easy-cosj100020-imaging +version: 1 +mode: assistant +difficulty: easy +datasets: + - dataset/imaging/cosj100020+015344 +workspace_packages: + - imaging +added: 2026-08-01 +--- + +# Benchmark: measure the bundled JWST galaxy's structure (assistant · easy) + +Almost everything this prompt asks for is directly available in the assistant: the dataset +ships with the repository, and data preparation, model composition, search configuration, +running the fit, plotting it and reading the result are all covered by existing core-loop +`ag_*` skills. A capable agent should complete it without inventing anything — the benchmark +measures whether it *finds and follows* the built-in workflow rather than writing +PyAutoGalaxy from memory. + +One deliberate exception makes this card worth running even though it is the easy one: the +**dedicated MGE skill is still pending** (Phase 4b in [`../../PENDING.md`](../../PENDING.md)). +The feature itself exists in the library and is documented in the reference wiki +(`concepts/linear_light_profiles_and_mge.md`, `api/light_profile_catalog.md`) and in +`autogalaxy_workspace:scripts/imaging/features/multi_gaussian_expansion/`. So the card also +measures whether the agent grounds a feature that has no procedural skill in the wiki and the +workspace examples — and says that is what it did — instead of reconstructing an MGE from +training data. + +## Prompt + +Paste verbatim as the first message of a fresh session (see +[`../AGENTS.md`](../AGENTS.md) for the run protocol): + +``` +Assistant mode. + +Fit the JWST F277W imaging in dataset/imaging/cosj100020+015344 with a multi-Gaussian +expansion (MGE) bulge, freeing the background sky level — the sky is not subtracted in +this data. Report the effective radius, axis ratio and position angle, add a single-Sersic +fit so I get a Sersic index to compare against, and then tell me how the recovered +structure changes across the other three wavebands. +``` + +This is Starter Prompt 2 of the top-level [`README.md`](../../README.md); the two texts must +stay identical (a divergence is a bug — fix the README or bump this card's `version`). +`autoassistant/tests/test_benchmark.py::test_repo_readme_prompts_match_cards` enforces it. + +## What this measures + +- **Routing**: does the agent use the assistant's skills, bundled dataset and reference wiki + rather than writing PyAutoGalaxy from memory? +- **The real-data safety gate**: plotting and inspecting the data, and settling both the + contaminant question and the mask extent, *before* composing any fit. +- **Reading the dataset's own caveats**: the un-subtracted sky is stated in the prompt, but + the faint 2.6" neighbour and the model-PSF systematic are not — they are in the dataset + README, and a good run finds them. +- **A complete run**: preparation → model → search → fit → the requested numbers, across + four wavebands. + +## Success rubric (100 points) + +### Machine-checkable (40) + +| # | Check | Pts | +|---|-------|-----| +| M1 | A script (or scripts) saved under `scripts/` that performs the fit | 5 | +| M2 | A completed non-linear search result exists under `output/` (not test-mode) | 10 | +| M3 | The model includes an MGE basis for the bulge **and** a free background sky level, both verifiable in the script | 10 | +| M4 | A single-Sersic comparison fit exists, with its own result under `output/` | 5 | +| M5 | A fit subplot figure was produced and its path shown to the user | 5 | +| M6 | The other three wavebands were fitted or measured, with per-band numbers reported | 5 | + +### Judged (60) + +| # | Criterion | Pts | +|---|-----------|-----| +| J1 | Real-data gate honoured: dataset plotted and inspected, and **both** questions settled before any fit — contaminants (the faint 2.6" neighbour) and the mask extent, with the chosen radius justified rather than left as a default | 15 | +| J2 | The sky pedestal is handled as the prompt asks and the agent explains the consequence of not doing so (it inflates the effective radius and Sersic index); the recovered level is sane against the dataset's measured value | 10 | +| J3 | Sensible model and priors for this galaxy; the MGE is grounded in the reference wiki or a workspace example and the agent says so, rather than recalled from memory | 10 | +| J4 | The requested quantities are all reported, and the agent is honest that `info.json`'s `effective_radius_arcsec_rough` is a prior-scale measurement rather than ground truth to be matched | 10 | +| J5 | The multi-band comparison is scientifically framed — the size trend with wavelength separated from the changing PSF width — and the shipped model PSF is acknowledged as the dominant systematic | 10 | +| J6 | Conduct: concise assistant-mode communication, no fabricated numbers, API-gate discipline (no invented symbols, functional `aplt` plotting) | 5 | + +## Operator notes + +- Expected wall-clock: roughly 15–60 minutes depending on hardware and the search chosen; + the non-linear searches dominate. Five fits are implied by the prompt (MGE + Sersic on + F277W, then three further bands), so an agent that proposes a cheaper route — a maximum + likelihood optimiser, or reusing priors across bands — is being sensible, not lazy. +- The galaxy is bright (peak S/N 180 at F277W) and smooth, so a poor fit is a modelling + problem rather than a data problem. A run that ends with the agent honestly reporting a + poor fit scores what the rubric gives it — record it; failures are data. +- Do not coach the agent toward the neighbour or the PSF caveat. Finding them is J1 and J5. diff --git a/dataset/imaging/cosj100020+015344/README.md b/dataset/imaging/cosj100020+015344/README.md new file mode 100644 index 0000000..81acbd9 --- /dev/null +++ b/dataset/imaging/cosj100020+015344/README.md @@ -0,0 +1,448 @@ +# COSJ100020+015344 — a four-band JWST/NIRCam galaxy-structure dataset + +A real multi-band JWST cutout of a **bright, resolved, non-lens** galaxy, reduced from +MAST level-2 exposures for use as the `autogalaxy_assistant` demonstration dataset: Sersic +and bulge+disk decomposition, multi-band modelling, and isophote / ellipse fitting. + +``` +README.md this file +info.json dataset-level summary (F277W values) +reduction_manifest.json per-band reduction + PSF provenance +wavebands/F115W/{data,noise_map,psf}.fits, info.json 419 x 419 @ 0.03"/pix (12.57") +wavebands/F150W/{data,noise_map,psf}.fits, info.json 419 x 419 @ 0.03"/pix (12.57") +wavebands/F277W/{data,noise_map,psf}.fits, info.json 209 x 209 @ 0.06"/pix (12.54") +wavebands/F444W/{data,noise_map,psf}.fits, info.json 209 x 209 @ 0.06"/pix (12.54") +``` + +Every `psf.fits` is a 21x21 unit-sum kernel. Data units are **MJy/sr** (native NIRCam +surface brightness — `calwebb_image3`'s own convention, not converted). Every cutout keeps +its WCS in the FITS header. + +That tree is the whole dataset. This README describes the reduction that produced it, so it +also refers to intermediate products that are deliberately **not** shipped: the 61x61 PSF +kernels, the rejected tier-1 empirical PSFs (§5b), the pipeline's own `reduction.json` / +`reduction_summary.json` / `psf_tier2b.json` records, the preview and selection figures, and +the working scripts (§9). Those are retained with the reduction run, not in this tree, and +nothing below depends on reading them: every number quoted in this README is either in the +prose itself, in `info.json`, or in `reduction_manifest.json`. + +**Read §5 before fitting.** Three properties of this data will silently bias a model that +ignores them: the sky is not subtracted, the PSF is a model PSF, and the short-wave depth +is not uniform across the cutout. + +## 1. What the target is + +| | | +|---|---| +| Name used here | **COSJ100020+015344** (the COWLS `COSJ` convention) | +| Position | RA = 150.08531°, Dec = +1.89582° (J2000) = 10:00:20.47 +01:53:44.9 | +| COSMOS2020 | Classic ID **499609** (Weaver et al. 2022, ApJS 258, 11; VizieR `J/ApJS/258/11/classic`) | +| zCOSMOS | ID **812167** | +| NED | `WISEA J100020.46+015344.7` | +| Redshift | **z = 0.3422**, spectroscopic — zCOSMOS-Bright DR3, confidence class 4.5, spectrum `zCOSMOS_BRIGHT_DR3_000812167_ZCMRa53_M2_Q4_17_1.fits` (Lilly et al. 2007, ApJS 172, 70; VizieR `J/ApJS/172/70/zcosmos3`). Independently z = 0.3426 in the zCOSMOS 20k group catalogue (Knobel et al. 2012, ApJ 753, 121; VizieR `J/ApJ/753/121`) | +| Catalogue photometry | Ks(AUTO) = 18.82, ACS F814W(AUTO) = 20.21, I = 20.21 | +| Catalogue stellar mass | log M\*/M☉ = 10.54 (COSMOS2020 LePhare median) / 10.64 (Knobel et al. 2012) | +| Morphology | smooth, centrally concentrated, mildly elongated — an early-type / bulge-dominated galaxy. COSMOS2020 records ACS axis ratio 0.92 and `ACSmuClass = 1` (galaxy) | +| Environment | member of zCOSMOS 20k group 379 at z ≈ 0.342. The next-brightest member (COSMOS2020 504922) is 16.5" away — outside the delivered cutout | +| Isolation | nearest catalogued neighbour 5.7". Measured on the delivered F277W cutout, the galaxy holds 989 of the 1043 detected flux units in the frame; the brightest neighbour is **8.0" north-west** (5% of the galaxy's flux) and a faint second sits **2.6"** away (0.3%) | + +**It is not a lens.** Cross-matched against the full COWLS catalogue +(, `catalogue.csv`, 440 entries +— the candidate list from COWLS I's visual inspection of the COSMOS-Web galaxy sample, +Nightingale et al. 2025, MNRAS 543, 203): the nearest COWLS entry is **55.5" away** — +`COSJ100024+015334`, the COSMOS-Web Ring itself, which was excluded from selection by +construction. NED lists no lens classification here. + +The cross-match was checked in both directions: it recovers the ring at 0.1" from its +published position, so a null result at this target is a real absence, not a coordinate +mismatch. + +## 2. How the target was selected + +The exposures already cached for the COSMOS-Web Ring reduction were reused (§3), so the +search was confined to the sky those exposures cover. + +1. **Footprint** — the sky area of the cached program-1727 `_cal` exposures was read from + their SCI-extension WCS: RA 150.039–150.139, Dec +1.802–+1.920, in two NIRCam module + pointings. `footprints.py` +2. **Scout mosaic** — one throwaway F277W mosaic was drizzled from the 8 module-B + exposures around the ring by calling `autoreduce`'s combine stage directly (381 s). + `scout_drizzle.py` +3. **Catalogue** — 914 COSMOS2020 Classic sources within 80" of the ring were pulled from + VizieR. Cuts: `lptype == 0` (LePhare galaxy classification), `ACSmuClass == 1` + (ACS star/galaxy class = galaxy), Ks(AUTO) < 20.5, nearest catalogued neighbour > 2". + `candidates.py` +4. **Coverage** — for each surviving candidate, the number of exposures covering a 15×15 + grid across a 13" box (the delivered cutout extent) was counted per band, using the + pipeline's own containment test. This rejects positions that look covered at the centre + but fall into a NIRCam short-wave detector gap at the edges. `cand_coverage_box.py` +5. **Visual inspection** — 13" F277W stamps of every candidate were rendered and inspected. + `inspect_scout.py` +6. **Lens exclusion** — the COWLS cross-match above. + +Coverage — exposures covering the 13" box, as min–median–max, with the COSMOS-Web Ring +position as a reference point: + +| | F115W | F150W | F277W | F444W | +|---|---|---|---|---| +| COSMOS-Web Ring (reference) | 2–4–6 | 2–4–6 | 1–4–4 | 1–4–4 | +| **COSJ100020+015344 (chosen)** | 2–4–4 | 2–4–4 | **4–4–4** | **4–4–4** | + +The chosen position is the brightest candidate whose *entire* long-wavelength cutout is +covered by all four exposures — better long-wavelength coverage than the ring dataset +itself. Brighter candidates were rejected on evidence: COSMOS2020 513945 (Ks = 17.4) falls +outside the cached footprint entirely, and 504922 (Ks = 18.1) sits in a 2-exposure region +with a weight edge crossing its cutout. + +Selection evidence was rendered as figures during the reduction run and is retained there: +the scout mosaic with the chosen galaxy marked, the excluded ring marked and every candidate +considered, plus a 13" zoom on the choice, and the 13" F277W stamp grid that the visual +inspection in step 5 was made on. Neither figure is part of the shipped dataset. + +## 3. Exposures and provenance of the inputs + +All science comes from **JWST GO program 1727, COSMOS-Web**, NIRCam imaging, level-2 +`_cal` products from MAST. No new download was made: the exposures were already in +`PyAutoReduce/scripts/cache/cosmos_web_ring_/` (fetched 2026-07-08/09 for the ring +integration test) and were **symlinked** into a scratch `ExposureCache` under this target's +names, with a manifest recording that origin (`seed_cache.py`). The CRDS reference store +was symlinked, never copied. Nothing under `PyAutoReduce/` was modified. + +Exposures entering each mosaic, after the pipeline's usability screen and +detector-footprint filter: + +| Band | Exposures combined | Detectors | Skipped off-target | Runtime | +|---|---|---|---|---| +| F115W | 12 | `nrcb1`–`nrcb4`, visit `jw01727138001`, `02101` dithers 1–4 | 52 | 522 s | +| F150W | 12 | `nrcb1`–`nrcb4`, visit `jw01727138001`, `04101` dithers 1–4 | 52 | 483 s | +| F277W | 4 | `nrcblong`, visit `jw01727138001`, `02101` dithers 1–4 | 12 | 197 s | +| F444W | 4 | `nrcblong`, visit `jw01727138001`, `04101` dithers 1–4 | 12 | 194 s | + +Total exposure time on the target: **1030.7 s** in every band. The exact exposure filenames +that entered each band ship here, in `reduction_manifest.json` under `.exposures`. The +pipeline's fuller per-band records (`reduction.json`, `reduction_summary.json`) are retained +with the reduction run. + +The short-wave counts are higher because a NIRCam short-wave module is four detectors and +each is its own `_cal` file: 4 dithers × the ~3 chips that come within the footprint +filter's margin of the target = 12 files. That is **not** extra depth — the number of +exposures actually covering any given point in the cutout is 2–4 (§2's coverage table), and +the step between those two values is exactly the non-uniformity of §5c. + +## 4. The pipeline + +`PyAutoReduce` (`autoreduce`) at commit **d7bd916a86c37b236bf3470b9bd5e43fcab28a62** +(`main`, 2026-07-29), driven by `reduce_cosj100020.py`, which mirrors the repository's own +`scripts/reduce_cosmos_web_ring.py`: same `TargetSpec` shape, same COSMOS-Web conventions +(short-wave 419×419 @ 0.03"/pix, long-wave 209×209 @ 0.06"/pix, `pixfrac = 1.0`, +`kernel = square`, program 1727 only). **No `autoreduce` source file was modified.** + +Stages: `acquire` (cache + CRDS + usability screen + footprint filter) → `align` +(a-priori WCS accepted, no TweakReg refinement pass) → `drizzle` (`calwebb_image3`: +tweakreg / skymatch / outlier_detection / resample, with `rotation = 0` and +`weight_type = ivm`) → `noise` → `psf` → `package`. + +Software: Python 3.12.10, `jwst` **2.0.1**, `astropy` 8.0.1, `numpy` 2.2.6, +`photutils` 2.3.0, `drizzlepac` 3.11.0, `astroquery` 0.4.11, `stpsf` 2.2.0, +`poppy` 1.1.2. CRDS context **`jwst_1535.pmap`** (JWST CRDS server). +No bad pixels required masking in any band (`n_masked_bad_pixels = 0` throughout). + +### Noise maps + +Read, not constructed: the `ERR` array `calwebb_image3`'s resample propagates +(Poisson + read noise + flat), multiplied by the Casertano correlated-noise factor **R** +that corrects for drizzle's pixel-to-pixel correlation. The pipeline's own blank-sky +consistency check (`sky_over_err_floor`) compares the empirical sky RMS against the `ERR` +floor — 1.0 means perfect agreement: + +| Band | R | `sky_over_err_floor` | median noise (MJy/sr) | +|---|---|---|---| +| F115W | 1.5254 | 1.095 | 0.0580 | +| F150W | 1.5254 | 1.017 | 0.0454 | +| F277W | 1.5384 | 0.961 | 0.0108 | +| F444W | 1.5384 | 0.880 | 0.0135 | + +The noise maps agree with the measured sky scatter to within 2–12%. F444W's 0.88 means the +noise map is ~12% *conservative* against the sky scatter there. + +## 5. Three things to know before you fit this data + +### 5a. The sky is **not** subtracted + +`calwebb_image3`'s skymatch step *matches* the exposures' backgrounds to each other but +does not subtract them, and `autoreduce` keeps the pipeline default. The delivered +`data.fits` therefore carries the real JWST sky as a positive pedestal, large compared +with the noise: + +| Band | background (MJy/sr) | median noise | pedestal / noise | +|---|---|---|---| +| F115W | 0.2624 | 0.0580 | 4.5 | +| F150W | 0.2551 | 0.0454 | 5.6 | +| F277W | 0.1246 | 0.0108 | 11.5 | +| F444W | 0.2614 | 0.0135 | 19.4 | + +**A light-profile fit that ignores this absorbs the pedestal into the Sersic wings** and +returns an inflated effective radius and Sersic index. Model it: + +```python +dataset_model = af.Model(ag.DatasetModel) +dataset_model.background_sky_level = af.UniformPrior(lower_limit=0.0, upper_limit=0.4) +model = af.Collection(galaxies=..., dataset_model=dataset_model) +``` + +or subtract the measured value (`info.json` → `background_sky_level`) first. The smoke fit +in §7 does the former and recovers the measured value. + +### 5b. The PSF is a **model** PSF, because the pipeline's empirical PSF is broken here + +This is the one place where the shipped products are not what `autoreduce`'s default path +produced. The reason is a real, measured defect that should be reported upstream. + +`autoreduce`'s tier-1 mosaic PSF builds an empirical ePSF from `DAOStarFinder` detections +on the drizzled mosaic. **In a deep JWST extragalactic field that selection is dominated +by compact galaxies, not stars.** Measured on this target's F277W scout mosaic +(`psf_diag.py`): + +* 327 candidates pass the star cuts and **only 24% are point-like**. The median half-light + radius of the selected "stars" is **3.2 native pixels** against **1.4** for a true F277W + point source; the 75th percentile is 5 px and the 95th is 13 px. +* The stacked kernel is therefore far too broad, and rings negative. Flux in the central + pixel, and the kernel half-light radius, against the COWLS COSMOS-Web reference kernels + for the same bands (`psf_table.py`): + + | Band | tier-1 central pixel | tier-1 r½ | COWLS reference central pixel | COWLS r½ | + |---|---|---|---|---| + | F115W | 1.7% | 0.222" | 16.2% | 0.042" | + | F150W | 4.5% | 0.153" | 14.3% | 0.042" | + | F277W | 0.6% | 0.537" | 14.1% | 0.085" | + | F444W | 1.2% | 0.379" | 9.0% | 0.085" | + + (F277W's r½ measured on the tier-1 61×61 kernel rather than the 21×21 box is 1.47", + ~17× the correct value; its minimum is −0.6 × its peak. Neither tier-1 kernel ships.) +* The failure reproduces on both the scout mosaic and the delivered reduction, in all four + bands — it is systematic, not one unlucky star list. It equally affects + `PyAutoReduce`'s own reference driver `scripts/reduce_cosmos_web_ring.py`, which runs the + identical code path on the identical field; that driver's acceptance check + (`autoreduce.validation.registered_ratios(new_data, new_noise, ref_data, ref_noise)`) + takes no PSF argument, which is why the defect has not been caught there. +* The other back-end the mosaic PSF stage offers, `TargetSpec(psf_backend="starred")` + (STARRED tier-1b), consumes the **same** star list, and **it fails the same way** — this + was measured, not assumed. Run on the identical 327-star list and scout mosaic, STARRED + returns a kernel with r½ = **1.11"** against tier-1's 1.445" and the correct ≈ 0.085": + 13× too broad, in 2436 s (41 min) for one band. Its own diagnostics report + `sampling_fwhm_px = 7.76` (0.47") and `undersampled: False` — it does not detect that + anything is wrong. **The defect is the star selection, not the back-end**, so the upstream + fix belongs in `autoreduce.psf.stars.find_stars` (a concentration cut — e.g. reject + candidates whose own half-light radius exceeds the point-source value by more than a small + margin), not in either PSF builder. + +Those kernels are **not shipped here.** The pipeline's tier-1 output was preserved with the +reduction run so the defect stays inspectable, and `reduction_manifest.json` records it per +band under `.psf_pipeline_tier1_rejected` — the star count, the measured kernel width, +and why it was rejected — so the rejection is auditable from what ships. + +The shipped `psf.fits` (and the 61×61 kernel that stays with the reduction run) are +instead the **tier-2b STPSF model PSF** that +`PyAutoReduce`'s own JWST design doc names as the JWST fallback tier +(`docs/design/jwst.md`: *"STPSF is the tier-2b fallback (per-detector, per-position)"*). +They are built by `psf_tier2b.py` entirely out of `autoreduce`'s own components — no new +algorithm is introduced: + +1. `autoreduce.psf.stpsf_model.model_frame_psf` — an STPSF model at each contributing + exposure's detector (`NRCBLONG`, `NRCB1`–`NRCB4`) and at the target's position on that + detector, from the `DET_DIST` extension (detector-sampled, geometric distortion + included); +2. `autoreduce.psf.frame_combine._drop_convolve` — convolution with the drizzle drop (the + `pixfrac = 1.0` box in native pixels); +3. `autoreduce.psf.frame_combine._resample_to_mosaic` — resampling onto the mosaic grid + through the local frame→mosaic WCS Jacobian at the target; +4. exposure-time-weighted average over the contributing exposures, then unit-normalised to + 21×21 and 61×61. Only the 21×21 kernel ships, as `psf.fits`; the 61×61 version is + retained with the reduction run. + +Steps 2–4 are exactly what `TargetSpec(psf_from_frames=True)` does; the only substitution +is the STPSF kernel for the galaxy-contaminated ePSF, which the mosaic PSF stage offers no +way to select. Per-band details — STPSF version, the contributing detectors and their +positions on each, the exposure weights, and the measured kernel FWHM and central-pixel +fraction — ship in `reduction_manifest.json` under `.psf`. + +**How good the shipped kernels are**, against the COWLS empirical reference: + +| Band | shipped central pixel | shipped r½ | COWLS central pixel | COWLS r½ | +|---|---|---|---|---| +| F115W | 24.7% | 0.030" | 16.2% | 0.042" | +| F150W | 18.3% | 0.037" | 14.3% | 0.042" | +| F277W | 18.4% | 0.074" | 14.1% | 0.085" | +| F444W | 10.2% | 0.085" | 9.0% | 0.085" | + +Physically sane, no negative ringing, and the right ordering with wavelength — but +**consistently sharper than the empirical reference**: the central-pixel fraction is ~50% +high in the most undersampled band (F115W) and converges to agreement in the best-sampled +one (F444W). (Compare the central-pixel fractions, not r½: for kernels this compact the r½ +estimator quantises onto the pixel grid — 0.030" and 0.042" at 0.03"/pix are adjacent grid +radii, not a resolved difference.) That is the expected +signature of a model PSF: STPSF does not carry the as-flown wavefront drift at the epoch of +these exposures, and the drop convolution plus local-affine resample capture drizzle's +broadening only to first order. **Treat the PSF as the dominant systematic in any fit to +this dataset**, and expect a fitted size to come out slightly *large* to compensate for a +slightly-too-sharp kernel — most so at F115W. + +### 5c. Coverage is uniform at long wavelength, stepped at short wavelength + +The pipeline records the drizzle weight uniformity over the delivered cutout as +`wht_rms_over_median`, against its own limit of 0.2: + +| Band | cutout weight uniformity | verdict | +|---|---|---| +| F115W | 0.286 | **exceeds the 0.2 limit** | +| F150W | 0.287 | **exceeds the 0.2 limit** | +| F277W | 0.085 | within limit | +| F444W | 0.079 | within limit | + +The short-wave 419×419 cutouts straddle a NIRCam short-wave detector-gap boundary: part of +the box has 4 exposures and part has 2, a √2 step in depth. The noise map tracks it +correctly (it is the propagated `ERR`), so this is a depth gradient, not an error. + +It is *not* the reason the short-wave peak S/N is lower (48 and 72, against F277W's 181): +that is mostly the 4× smaller pixel area at 0.03"/pix collecting 4× fewer photons per +pixel, plus the galaxy being intrinsically fainter blueward at z = 0.34. The depth step is a +gradient across the frame, not an overall sensitivity loss. + +A modelling mask of radius ≲ 4" keeps the fit inside the uniform region and excludes the +8.0" neighbour of §1. It does **not** exclude the faint source 2.6" from the centre — that +one has to be masked explicitly or modelled (it is visible in the smoke fit's residual +map, §7). + +## 6. `info.json` — where every number comes from + +`info.json` at the dataset root, and one per band under `wavebands//`. All of them +were written by the reduction run's `measure.py` (§9), which is the only place these numbers +are produced. +**Nothing is copied from a catalogue and nothing is estimated by eye.** + +| Key | Origin | +|---|---| +| `pixel_scale` | The reduction's `package.pixel_scale` — the `TargetSpec.final_scale` the instrument adapter recommends (0.03 short-wave, 0.06 long-wave) | +| `centre_ra`, `centre_dec` | Flux-weighted centroid of the segmentation-detected central source on the background-subtracted `data.fits`, through the cutout WCS | +| `axis_ratio` | `semiminor_sigma / semimajor_sigma` from the source's second-order central moments (`photutils.morphology.data_properties`), measured over that source's own segment only so a neighbour cannot pull the shape | +| `position_angle` | Major-axis angle from the same moments, counter-clockwise from +x, wrapped to [0, 180) | +| `effective_radius_arcsec_rough` | **Rough, and biased low.** Radius of the elliptical aperture (at the measured axis ratio and angle) enclosing half the flux inside a 3" truncation. A truncated curve-of-growth half-light radius, **not** a Sersic `effective_radius`: the aperture is truncated and nothing extrapolates the wings. Published as a scale for priors | +| `moment_semimajor_sigma_arcsec` | Second-moment semimajor sigma, same measurement | +| `background_sky_level`, `background_sky_rms` | Sigma-clipped median and standard deviation of `data.fits` outside every detected segment — the un-subtracted sky of §5a | +| `peak_snr`, `peak_value` | Max of `data / noise_map`, and of the background-subtracted data, over the source segment | +| `segment_area_pixels`, `curve_of_growth_max_arcsec` | Bookkeeping for the two measurements above | +| `band_to_band_scatter` | Standard deviation of `axis_ratio` and `effective_radius_arcsec_rough` across the four bands. **This is the honest error bar on those numbers** | +| redshift | **Not measured, and not in `info.json` as a measurement.** z = 0.3422 is a published catalogue value, cited in §1 | + +Measured values, all four bands: + +| | F115W | F150W | F277W | F444W | +|---|---|---|---|---| +| centre RA | 150.085317 | 150.085321 | 150.085325 | 150.085320 | +| centre Dec | +1.895805 | +1.895825 | +1.895828 | +1.895825 | +| axis ratio | 0.886 | 0.849 | 0.834 | 0.872 | +| position angle | 98.8° | 93.1° | 95.1° | 89.8° | +| rough effective radius | 0.297" | 0.288" | 0.370" | 0.373" | +| peak S/N | 47.8 | 71.7 | 180.0 | 124.2 | + +The dataset-level `info.json` promotes the **F277W** values (highest S/N, uniform +coverage). The four bands agree on the centre to **0.08"**, on the axis ratio to 0.05, and +on the position angle to 9°; the rough size steps from ≈ 0.29" short-ward to ≈ 0.37" +long-ward, which is a genuine wavelength dependence mixed with the PSF difference, not a +measurement error. The measured F277W centre is **0.02" from the COSMOS2020 catalogue +position** — an independent check that the reduction centred where it claims to. + +No magnitudes are published here. The surface-brightness-to-AB conversion, aperture +correction and Galactic-extinction treatment needed to do that honestly were not carried +out; use the COSMOS2020 photometry cited in §1 if you need magnitudes. + +## 7. Validation + +* **Loads** — every band loads through `ag.Imaging.from_fits` at its own pixel scale with a + unit-sum PSF, at the shapes listed at the top of this file. The per-band dataset subplots + this was checked on are retained with the reduction run. (`validate.py`) +* **Smoke fit** — a single `ag.lp.Sersic` plus a free + `ag.DatasetModel.background_sky_level`, maximum likelihood via `af.LBFGS`, on the F277W + data inside a 3" circular mask (7825 pixels). It converges: + + | | fitted | start point | independent value | + |---|---|---|---| + | log likelihood | **+23988** | −31824 | | + | reduced χ² | **0.797** | | | + | normalised residual RMS | **0.893** | | | + | centre offset from cutout centre | (−0.054", −0.032") | (0, 0) | ≤ 1 pixel | + | axis ratio | 0.916 | 0.834 | 0.83–0.89 across bands (§6) | + | position angle | 89.5° | 95.1° | 89.8–98.8° across bands (§6) | + | effective radius | 0.351" | 0.370" | 0.370" rough measurement (§6) | + | Sersic index | 4.697 | 4.0 | — | + | background sky level | 0.12278 | 0.12458 | 0.12458 measured (§5a) | + + The free sky recovers the independently measured pedestal to **1.4%**, the reduced χ² of + 0.80 says the noise map is if anything slightly conservative (consistent with §4's + `sky_over_err_floor`), and n ≈ 4.7 is a de Vaucouleurs-like early type, which is what + the image looks like. The fit's result files and residual figure are retained with the + reduction run; the table above is the record of it that ships. (`smoke_fit.py`) + + The residual map is flat except for a few-pixel dipole at the very centre (the usual + PSF/centring residual on a peak-S/N-181 core) and the faint neighbour 2.6" from the + centre, which sits inside the 3" mask and which a single-Sersic model has nothing to + represent. + + Two implementation notes, because both changed the answer: + * The optimiser is started from `measure.py`'s moment measurements, with the start + `intensity` obtained by rendering a unit-intensity trial profile through the dataset's + own PSF and grid and matching its total flux to the data. Started from a random prior + draw instead, L-BFGS-B walks into the prior corner (`effective_radius` pinned at its + lower limit, reduced χ² = 140) and reports a "converged" fit that is nonsense. + * With JAX enabled, L-BFGS-B terminates at iteration **zero** and returns its start + point unchanged (reduced χ² = 15.1) while still logging "Search complete". The fit + above needs `PYAUTO_DISABLE_JAX=1`. That is worth chasing separately — it is not a + property of this dataset. + + This is a **smoke test, not a science fit**: no sampler, no error bars, no + PSF-systematic budget, and §5b applies in full. It exists to show that the data, the + noise map and the PSF are mutually consistent and that a light profile converges on + them. + +## 8. What is deliberately *not* claimed + +* No magnitudes, stellar mass, or physical size in kpc measured from this data (§6). +* No Sersic `effective_radius`, index or ellipticity presented as a measurement — the + smoke fit's values are a smoke fit's values, and the published `effective_radius` is + labelled `_rough` because it is. +* No redshift beyond the published catalogue value, which is cited rather than derived. +* No claim that the PSF is empirical (§5b), that the sky is subtracted (§5a), or that the + short-wave depth is uniform (§5c). + +## 9. Reproducing this + +The scripts that produced this dataset are **working scripts, not part of the dataset**, and +are retained with the reduction run rather than shipped here. They expect to run from the +working directory they were written in (they resolve the exposure cache and the scout mosaic +relative to their own location) and they import `autoreduce` by `sys.path`, exactly as +`PyAutoReduce`'s own `scripts/` do. They are listed so the provenance chain above is legible +end to end, in the order they must run: + +``` +seed_cache.py symlink the cached program-1727 exposures under this target's names +footprints.py sky area of the cached exposures, per band +scout_drizzle.py one throwaway F277W mosaic for the target search +candidates.py COSMOS2020 shortlist +cand_coverage.py per-band coverage at a list of candidate positions +cand_coverage_all.py per-band coverage of every catalogue source +cand_coverage_box.py per-band coverage across the 13" cutout box +inspect_scout.py candidate stamps + field figure +psf_diag.py what the tier-1 star selection actually selected (§5b) +psf_backend_test.py tier-1 vs STARRED on the same star list (§5b) +reduce_cosj100020.py THE REDUCTION — --band F115W|F150W|F277W|F444W|all +assemble.py copy the shipping products into this directory +psf_tier2b.py the tier-2b STPSF PSFs, replacing tier-1 (§5b) +psf_table.py the PSF comparison tables in §5b +measure.py info.json +selection_figure.py preview/target_selection.png +validate.py load every band + preview subplots +smoke_fit.py the F277W smoke fit +``` + +`smoke_fit.py` needs `PYAUTO_DISABLE_JAX=1`: with JAX enabled, L-BFGS-B terminates at +iteration zero and returns its own start point unchanged. diff --git a/dataset/imaging/cosj100020+015344/info.json b/dataset/imaging/cosj100020+015344/info.json new file mode 100644 index 0000000..c289a8e --- /dev/null +++ b/dataset/imaging/cosj100020+015344/info.json @@ -0,0 +1,82 @@ +{ + "data_name": "COSJ100020+015344", + "reference_band": "F277W", + "centre_ra": 150.085325, + "centre_dec": 1.895828, + "axis_ratio": 0.834, + "position_angle": 95.1, + "effective_radius_arcsec_rough": 0.37, + "band_to_band_scatter": { + "axis_ratio": 0.02, + "effective_radius_arcsec_rough": 0.04 + }, + "background_sky_level_per_band": { + "F115W": 0.26236, + "F150W": 0.25509, + "F277W": 0.12458, + "F444W": 0.26144 + }, + "measurement_note": "centre / axis_ratio / position_angle are second-moment measurements of the delivered F277W data.fits; effective_radius_arcsec_rough is a half-light radius from an elliptical curve of growth truncated at 3.0\" and is biased low \u2014 it is a scale for priors, not a Sersic effective radius. See README.md.", + "per_band": { + "F115W": { + "pixel_scale": 0.03, + "centre_ra": 150.085317, + "centre_dec": 1.895805, + "axis_ratio": 0.886, + "position_angle": 98.8, + "effective_radius_arcsec_rough": 0.297, + "moment_semimajor_sigma_arcsec": 0.285, + "peak_snr": 47.8, + "peak_value": 24.2005, + "background_sky_level": 0.26236, + "background_sky_rms": 0.0332, + "curve_of_growth_max_arcsec": 3.0, + "segment_area_pixels": 2756 + }, + "F150W": { + "pixel_scale": 0.03, + "centre_ra": 150.085321, + "centre_dec": 1.895825, + "axis_ratio": 0.849, + "position_angle": 93.1, + "effective_radius_arcsec_rough": 0.288, + "moment_semimajor_sigma_arcsec": 0.308, + "peak_snr": 71.7, + "peak_value": 46.6986, + "background_sky_level": 0.25509, + "background_sky_rms": 0.02461, + "curve_of_growth_max_arcsec": 3.0, + "segment_area_pixels": 3794 + }, + "F277W": { + "pixel_scale": 0.06, + "centre_ra": 150.085325, + "centre_dec": 1.895828, + "axis_ratio": 0.834, + "position_angle": 95.1, + "effective_radius_arcsec_rough": 0.37, + "moment_semimajor_sigma_arcsec": 0.523, + "peak_snr": 180.0, + "peak_value": 24.3112, + "background_sky_level": 0.12458, + "background_sky_rms": 0.00568, + "curve_of_growth_max_arcsec": 3.0, + "segment_area_pixels": 3659 + }, + "F444W": { + "pixel_scale": 0.06, + "centre_ra": 150.08532, + "centre_dec": 1.895825, + "axis_ratio": 0.872, + "position_angle": 89.8, + "effective_radius_arcsec_rough": 0.373, + "moment_semimajor_sigma_arcsec": 0.45, + "peak_snr": 124.2, + "peak_value": 9.7187, + "background_sky_level": 0.26144, + "background_sky_rms": 0.00634, + "curve_of_growth_max_arcsec": 3.0, + "segment_area_pixels": 2321 + } + } +} diff --git a/dataset/imaging/cosj100020+015344/reduction_manifest.json b/dataset/imaging/cosj100020+015344/reduction_manifest.json new file mode 100644 index 0000000..91e65e5 --- /dev/null +++ b/dataset/imaging/cosj100020+015344/reduction_manifest.json @@ -0,0 +1,354 @@ +{ + "F115W": { + "pixel_scale": 0.03, + "cutout_shape": [ + 419, + 419 + ], + "data_units": "MJy/sr", + "n_exposures": 12, + "exposures": [ + "jw01727138001_02101_00001_nrcb1_cal.fits", + "jw01727138001_02101_00001_nrcb2_cal.fits", + "jw01727138001_02101_00001_nrcb3_cal.fits", + "jw01727138001_02101_00001_nrcb4_cal.fits", + "jw01727138001_02101_00002_nrcb1_cal.fits", + "jw01727138001_02101_00002_nrcb2_cal.fits", + "jw01727138001_02101_00002_nrcb3_cal.fits", + "jw01727138001_02101_00002_nrcb4_cal.fits", + "jw01727138001_02101_00003_nrcb1_cal.fits", + "jw01727138001_02101_00003_nrcb2_cal.fits", + "jw01727138001_02101_00004_nrcb1_cal.fits", + "jw01727138001_02101_00004_nrcb2_cal.fits" + ], + "n_skipped_off_target": 52, + "correlated_noise_factor": 1.5253968253968253, + "noise_recipe": "R * ERR (propagated by calwebb_image3 resample)", + "weight_uniformity_cutout": { + "wht_rms_over_median": 0.28581055998802185, + "limit": 0.2, + "acceptable": false + }, + "n_masked_bad_pixels": 0, + "psf": { + "method": "stpsf-tier2b-mosaic", + "shipped_as": "psf.fits", + "kernel_shape": [ + 21, + 21 + ], + "normalisation": "unit sum", + "stpsf_version": "2.2.0", + "ext": "DET_DIST (detector-sampled, geometric distortion included)", + "n_frames_combined": 2, + "detectors": [ + "NRCB1" + ], + "weighting": "exptime", + "pixfrac_drop_convolved": 1.0, + "resample": "local-affine frame->mosaic Jacobian at the target", + "moment_fwhm_pix": 4.967852165977171, + "central_pixel_flux_fraction": 0.24691423891227005, + "frames": [ + { + "exposure": "jw01727138001_02101_00003_nrcb1_cal.fits", + "chip": 1, + "weight_exptime": 257.682, + "detector": "NRCB1", + "detector_position": [ + 1580.7435433103335, + 462.9434393821838 + ] + }, + { + "exposure": "jw01727138001_02101_00004_nrcb1_cal.fits", + "chip": 1, + "weight_exptime": 257.682, + "detector": "NRCB1", + "detector_position": [ + 1584.8938023187532, + 268.6342388443356 + ] + } + ], + "caveat": "model PSF: no as-flown wavefront drift for this epoch; drop convolution + local-affine resample are first-order. The PSF is the dominant systematic in any fit to this dataset." + }, + "psf_pipeline_tier1_rejected": { + "method": "epsf-tier1", + "n_stars_used": 72, + "oversampling": 2, + "fwhm_pix": 4.47213595499958, + "reason_rejected": "The pipeline's tier-1 mosaic ePSF stage ran and produced this kernel, but it is NOT the shipped psf.fits. Its DAOStarFinder star list on this deep extragalactic field is ~76% resolved compact galaxies rather than stars, so the stacked kernel is far too broad and rings negative. The rejected kernels are not shipped with this dataset; they are retained with the reduction run. See README.md section 5b." + } + }, + "F150W": { + "pixel_scale": 0.03, + "cutout_shape": [ + 419, + 419 + ], + "data_units": "MJy/sr", + "n_exposures": 12, + "exposures": [ + "jw01727138001_04101_00001_nrcb1_cal.fits", + "jw01727138001_04101_00001_nrcb2_cal.fits", + "jw01727138001_04101_00001_nrcb3_cal.fits", + "jw01727138001_04101_00001_nrcb4_cal.fits", + "jw01727138001_04101_00002_nrcb1_cal.fits", + "jw01727138001_04101_00002_nrcb2_cal.fits", + "jw01727138001_04101_00002_nrcb3_cal.fits", + "jw01727138001_04101_00002_nrcb4_cal.fits", + "jw01727138001_04101_00003_nrcb1_cal.fits", + "jw01727138001_04101_00003_nrcb2_cal.fits", + "jw01727138001_04101_00004_nrcb1_cal.fits", + "jw01727138001_04101_00004_nrcb2_cal.fits" + ], + "n_skipped_off_target": 52, + "correlated_noise_factor": 1.5253968253968253, + "noise_recipe": "R * ERR (propagated by calwebb_image3 resample)", + "weight_uniformity_cutout": { + "wht_rms_over_median": 0.2866322994232178, + "limit": 0.2, + "acceptable": false + }, + "n_masked_bad_pixels": 0, + "psf": { + "method": "stpsf-tier2b-mosaic", + "shipped_as": "psf.fits", + "kernel_shape": [ + 21, + 21 + ], + "normalisation": "unit sum", + "stpsf_version": "2.2.0", + "ext": "DET_DIST (detector-sampled, geometric distortion included)", + "n_frames_combined": 2, + "detectors": [ + "NRCB1" + ], + "weighting": "exptime", + "pixfrac_drop_convolved": 1.0, + "resample": "local-affine frame->mosaic Jacobian at the target", + "moment_fwhm_pix": 4.8803662334164395, + "central_pixel_flux_fraction": 0.18264542103264655, + "frames": [ + { + "exposure": "jw01727138001_04101_00003_nrcb1_cal.fits", + "chip": 1, + "weight_exptime": 257.682, + "detector": "NRCB1", + "detector_position": [ + 1580.75013686257, + 462.9224621938993 + ] + }, + { + "exposure": "jw01727138001_04101_00004_nrcb1_cal.fits", + "chip": 1, + "weight_exptime": 257.682, + "detector": "NRCB1", + "detector_position": [ + 1584.899647642625, + 268.6239693053857 + ] + } + ], + "caveat": "model PSF: no as-flown wavefront drift for this epoch; drop convolution + local-affine resample are first-order. The PSF is the dominant systematic in any fit to this dataset." + }, + "psf_pipeline_tier1_rejected": { + "method": "epsf-tier1", + "n_stars_used": 89, + "oversampling": 2, + "fwhm_pix": 2.8284271247461903, + "reason_rejected": "The pipeline's tier-1 mosaic ePSF stage ran and produced this kernel, but it is NOT the shipped psf.fits. Its DAOStarFinder star list on this deep extragalactic field is ~76% resolved compact galaxies rather than stars, so the stacked kernel is far too broad and rings negative. The rejected kernels are not shipped with this dataset; they are retained with the reduction run. See README.md section 5b." + } + }, + "F277W": { + "pixel_scale": 0.06, + "cutout_shape": [ + 209, + 209 + ], + "data_units": "MJy/sr", + "n_exposures": 4, + "exposures": [ + "jw01727138001_02101_00001_nrcblong_cal.fits", + "jw01727138001_02101_00002_nrcblong_cal.fits", + "jw01727138001_02101_00003_nrcblong_cal.fits", + "jw01727138001_02101_00004_nrcblong_cal.fits" + ], + "n_skipped_off_target": 12, + "correlated_noise_factor": 1.538372093023256, + "noise_recipe": "R * ERR (propagated by calwebb_image3 resample)", + "weight_uniformity_cutout": { + "wht_rms_over_median": 0.08479592949151993, + "limit": 0.2, + "acceptable": true + }, + "n_masked_bad_pixels": 0, + "psf": { + "method": "stpsf-tier2b-mosaic", + "shipped_as": "psf.fits", + "kernel_shape": [ + 21, + 21 + ], + "normalisation": "unit sum", + "stpsf_version": "2.2.0", + "ext": "DET_DIST (detector-sampled, geometric distortion included)", + "n_frames_combined": 4, + "detectors": [ + "NRCBLONG" + ], + "weighting": "exptime", + "pixfrac_drop_convolved": 1.0, + "resample": "local-affine frame->mosaic Jacobian at the target", + "moment_fwhm_pix": 4.570972517804071, + "central_pixel_flux_fraction": 0.1838074555167187, + "frames": [ + { + "exposure": "jw01727138001_02101_00001_nrcblong_cal.fits", + "chip": 1, + "weight_exptime": 257.682, + "detector": "NRCBLONG", + "detector_position": [ + 1040.8394488727095, + 1187.9808875733115 + ] + }, + { + "exposure": "jw01727138001_02101_00002_nrcblong_cal.fits", + "chip": 1, + "weight_exptime": 257.682, + "detector": "NRCBLONG", + "detector_position": [ + 1037.7469047561729, + 1283.292762500006 + ] + }, + { + "exposure": "jw01727138001_02101_00003_nrcblong_cal.fits", + "chip": 1, + "weight_exptime": 257.682, + "detector": "NRCBLONG", + "detector_position": [ + 1826.4067785504917, + 1281.4274656083066 + ] + }, + { + "exposure": "jw01727138001_02101_00004_nrcblong_cal.fits", + "chip": 1, + "weight_exptime": 257.682, + "detector": "NRCBLONG", + "detector_position": [ + 1828.66204067103, + 1185.9547221500577 + ] + } + ], + "caveat": "model PSF: no as-flown wavefront drift for this epoch; drop convolution + local-affine resample are first-order. The PSF is the dominant systematic in any fit to this dataset." + }, + "psf_pipeline_tier1_rejected": { + "method": "epsf-tier1", + "n_stars_used": 154, + "oversampling": 2, + "fwhm_pix": 0.0, + "reason_rejected": "The pipeline's tier-1 mosaic ePSF stage ran and produced this kernel, but it is NOT the shipped psf.fits. Its DAOStarFinder star list on this deep extragalactic field is ~76% resolved compact galaxies rather than stars, so the stacked kernel is far too broad and rings negative. The rejected kernels are not shipped with this dataset; they are retained with the reduction run. See README.md section 5b." + } + }, + "F444W": { + "pixel_scale": 0.06, + "cutout_shape": [ + 209, + 209 + ], + "data_units": "MJy/sr", + "n_exposures": 4, + "exposures": [ + "jw01727138001_04101_00001_nrcblong_cal.fits", + "jw01727138001_04101_00002_nrcblong_cal.fits", + "jw01727138001_04101_00003_nrcblong_cal.fits", + "jw01727138001_04101_00004_nrcblong_cal.fits" + ], + "n_skipped_off_target": 12, + "correlated_noise_factor": 1.538372093023256, + "noise_recipe": "R * ERR (propagated by calwebb_image3 resample)", + "weight_uniformity_cutout": { + "wht_rms_over_median": 0.07922098785638809, + "limit": 0.2, + "acceptable": true + }, + "n_masked_bad_pixels": 0, + "psf": { + "method": "stpsf-tier2b-mosaic", + "shipped_as": "psf.fits", + "kernel_shape": [ + 21, + 21 + ], + "normalisation": "unit sum", + "stpsf_version": "2.2.0", + "ext": "DET_DIST (detector-sampled, geometric distortion included)", + "n_frames_combined": 4, + "detectors": [ + "NRCBLONG" + ], + "weighting": "exptime", + "pixfrac_drop_convolved": 1.0, + "resample": "local-affine frame->mosaic Jacobian at the target", + "moment_fwhm_pix": 5.335327502270906, + "central_pixel_flux_fraction": 0.10184916347006297, + "frames": [ + { + "exposure": "jw01727138001_04101_00001_nrcblong_cal.fits", + "chip": 1, + "weight_exptime": 257.682, + "detector": "NRCBLONG", + "detector_position": [ + 1040.9046447059848, + 1188.330174567066 + ] + }, + { + "exposure": "jw01727138001_04101_00002_nrcblong_cal.fits", + "chip": 1, + "weight_exptime": 257.682, + "detector": "NRCBLONG", + "detector_position": [ + 1037.8309514065854, + 1283.615469705321 + ] + }, + { + "exposure": "jw01727138001_04101_00003_nrcblong_cal.fits", + "chip": 1, + "weight_exptime": 257.682, + "detector": "NRCBLONG", + "detector_position": [ + 1826.4409614311226, + 1281.7350505543343 + ] + }, + { + "exposure": "jw01727138001_04101_00004_nrcblong_cal.fits", + "chip": 1, + "weight_exptime": 257.682, + "detector": "NRCBLONG", + "detector_position": [ + 1828.6940367692023, + 1186.2695027529182 + ] + } + ], + "caveat": "model PSF: no as-flown wavefront drift for this epoch; drop convolution + local-affine resample are first-order. The PSF is the dominant systematic in any fit to this dataset." + }, + "psf_pipeline_tier1_rejected": { + "method": "epsf-tier1", + "n_stars_used": 130, + "oversampling": 2, + "fwhm_pix": 6.324555320336759, + "reason_rejected": "The pipeline's tier-1 mosaic ePSF stage ran and produced this kernel, but it is NOT the shipped psf.fits. Its DAOStarFinder star list on this deep extragalactic field is ~76% resolved compact galaxies rather than stars, so the stacked kernel is far too broad and rings negative. The rejected kernels are not shipped with this dataset; they are retained with the reduction run. See README.md section 5b." + } + } +} diff --git a/dataset/imaging/cosj100020+015344/wavebands/F115W/data.fits b/dataset/imaging/cosj100020+015344/wavebands/F115W/data.fits new file mode 100644 index 0000000..1fdc039 Binary files /dev/null and b/dataset/imaging/cosj100020+015344/wavebands/F115W/data.fits differ diff --git a/dataset/imaging/cosj100020+015344/wavebands/F115W/info.json b/dataset/imaging/cosj100020+015344/wavebands/F115W/info.json new file mode 100644 index 0000000..40d5424 --- /dev/null +++ b/dataset/imaging/cosj100020+015344/wavebands/F115W/info.json @@ -0,0 +1,15 @@ +{ + "pixel_scale": 0.03, + "centre_ra": 150.085317, + "centre_dec": 1.895805, + "axis_ratio": 0.886, + "position_angle": 98.8, + "effective_radius_arcsec_rough": 0.297, + "moment_semimajor_sigma_arcsec": 0.285, + "peak_snr": 47.8, + "peak_value": 24.2005, + "background_sky_level": 0.26236, + "background_sky_rms": 0.0332, + "curve_of_growth_max_arcsec": 3.0, + "segment_area_pixels": 2756 +} diff --git a/dataset/imaging/cosj100020+015344/wavebands/F115W/noise_map.fits b/dataset/imaging/cosj100020+015344/wavebands/F115W/noise_map.fits new file mode 100644 index 0000000..98796c0 Binary files /dev/null and b/dataset/imaging/cosj100020+015344/wavebands/F115W/noise_map.fits differ diff --git a/dataset/imaging/cosj100020+015344/wavebands/F115W/psf.fits b/dataset/imaging/cosj100020+015344/wavebands/F115W/psf.fits new file mode 100644 index 0000000..b16bd62 Binary files /dev/null and b/dataset/imaging/cosj100020+015344/wavebands/F115W/psf.fits differ diff --git a/dataset/imaging/cosj100020+015344/wavebands/F150W/data.fits b/dataset/imaging/cosj100020+015344/wavebands/F150W/data.fits new file mode 100644 index 0000000..9a59051 Binary files /dev/null and b/dataset/imaging/cosj100020+015344/wavebands/F150W/data.fits differ diff --git a/dataset/imaging/cosj100020+015344/wavebands/F150W/info.json b/dataset/imaging/cosj100020+015344/wavebands/F150W/info.json new file mode 100644 index 0000000..c26a43b --- /dev/null +++ b/dataset/imaging/cosj100020+015344/wavebands/F150W/info.json @@ -0,0 +1,15 @@ +{ + "pixel_scale": 0.03, + "centre_ra": 150.085321, + "centre_dec": 1.895825, + "axis_ratio": 0.849, + "position_angle": 93.1, + "effective_radius_arcsec_rough": 0.288, + "moment_semimajor_sigma_arcsec": 0.308, + "peak_snr": 71.7, + "peak_value": 46.6986, + "background_sky_level": 0.25509, + "background_sky_rms": 0.02461, + "curve_of_growth_max_arcsec": 3.0, + "segment_area_pixels": 3794 +} diff --git a/dataset/imaging/cosj100020+015344/wavebands/F150W/noise_map.fits b/dataset/imaging/cosj100020+015344/wavebands/F150W/noise_map.fits new file mode 100644 index 0000000..f7f4c7e Binary files /dev/null and b/dataset/imaging/cosj100020+015344/wavebands/F150W/noise_map.fits differ diff --git a/dataset/imaging/cosj100020+015344/wavebands/F150W/psf.fits b/dataset/imaging/cosj100020+015344/wavebands/F150W/psf.fits new file mode 100644 index 0000000..0a9b432 Binary files /dev/null and b/dataset/imaging/cosj100020+015344/wavebands/F150W/psf.fits differ diff --git a/dataset/imaging/cosj100020+015344/wavebands/F277W/data.fits b/dataset/imaging/cosj100020+015344/wavebands/F277W/data.fits new file mode 100644 index 0000000..605bf1b Binary files /dev/null and b/dataset/imaging/cosj100020+015344/wavebands/F277W/data.fits differ diff --git a/dataset/imaging/cosj100020+015344/wavebands/F277W/info.json b/dataset/imaging/cosj100020+015344/wavebands/F277W/info.json new file mode 100644 index 0000000..272365a --- /dev/null +++ b/dataset/imaging/cosj100020+015344/wavebands/F277W/info.json @@ -0,0 +1,15 @@ +{ + "pixel_scale": 0.06, + "centre_ra": 150.085325, + "centre_dec": 1.895828, + "axis_ratio": 0.834, + "position_angle": 95.1, + "effective_radius_arcsec_rough": 0.37, + "moment_semimajor_sigma_arcsec": 0.523, + "peak_snr": 180.0, + "peak_value": 24.3112, + "background_sky_level": 0.12458, + "background_sky_rms": 0.00568, + "curve_of_growth_max_arcsec": 3.0, + "segment_area_pixels": 3659 +} diff --git a/dataset/imaging/cosj100020+015344/wavebands/F277W/noise_map.fits b/dataset/imaging/cosj100020+015344/wavebands/F277W/noise_map.fits new file mode 100644 index 0000000..18d7fec Binary files /dev/null and b/dataset/imaging/cosj100020+015344/wavebands/F277W/noise_map.fits differ diff --git a/dataset/imaging/cosj100020+015344/wavebands/F277W/psf.fits b/dataset/imaging/cosj100020+015344/wavebands/F277W/psf.fits new file mode 100644 index 0000000..4f80b89 Binary files /dev/null and b/dataset/imaging/cosj100020+015344/wavebands/F277W/psf.fits differ diff --git a/dataset/imaging/cosj100020+015344/wavebands/F444W/data.fits b/dataset/imaging/cosj100020+015344/wavebands/F444W/data.fits new file mode 100644 index 0000000..86604b4 Binary files /dev/null and b/dataset/imaging/cosj100020+015344/wavebands/F444W/data.fits differ diff --git a/dataset/imaging/cosj100020+015344/wavebands/F444W/info.json b/dataset/imaging/cosj100020+015344/wavebands/F444W/info.json new file mode 100644 index 0000000..e9b704e --- /dev/null +++ b/dataset/imaging/cosj100020+015344/wavebands/F444W/info.json @@ -0,0 +1,15 @@ +{ + "pixel_scale": 0.06, + "centre_ra": 150.08532, + "centre_dec": 1.895825, + "axis_ratio": 0.872, + "position_angle": 89.8, + "effective_radius_arcsec_rough": 0.373, + "moment_semimajor_sigma_arcsec": 0.45, + "peak_snr": 124.2, + "peak_value": 9.7187, + "background_sky_level": 0.26144, + "background_sky_rms": 0.00634, + "curve_of_growth_max_arcsec": 3.0, + "segment_area_pixels": 2321 +} diff --git a/dataset/imaging/cosj100020+015344/wavebands/F444W/noise_map.fits b/dataset/imaging/cosj100020+015344/wavebands/F444W/noise_map.fits new file mode 100644 index 0000000..820fd5f Binary files /dev/null and b/dataset/imaging/cosj100020+015344/wavebands/F444W/noise_map.fits differ diff --git a/dataset/imaging/cosj100020+015344/wavebands/F444W/psf.fits b/dataset/imaging/cosj100020+015344/wavebands/F444W/psf.fits new file mode 100644 index 0000000..4e39202 Binary files /dev/null and b/dataset/imaging/cosj100020+015344/wavebands/F444W/psf.fits differ diff --git a/docs/images/cosj100020+015344_dataset.png b/docs/images/cosj100020+015344_dataset.png new file mode 100644 index 0000000..7bf1ee6 Binary files /dev/null and b/docs/images/cosj100020+015344_dataset.png differ diff --git a/docs/images/sources/README.md b/docs/images/sources/README.md new file mode 100644 index 0000000..9151f1c --- /dev/null +++ b/docs/images/sources/README.md @@ -0,0 +1,46 @@ +# Figure sources + +Provenance of the images under `docs/images/`. The rule this directory exists to record: +**every committed figure must be rebuildable offline from material already in this +repository.** + +## Vendored third-party inputs + +**None.** This directory holds no image assets, and that is deliberate rather than an +oversight: the only README figure is built from the bundled dataset's own FITS files, so +there is no survey cutout, colour composite or externally-published image to vendor. If a +future figure does need a third-party asset, vendor it here and add a row to the table +below — never fetch it at build time, because the figure scripts must run without network +access. + +| File | Source | Notes | +|------|--------|-------| +| _(none)_ | | | + +## Generated figures + +| File | Built by | Inputs | +|------|----------|--------| +| `../cosj100020+015344_dataset.png` | [`../../make_readme_figures.py`](../../make_readme_figures.py) | `dataset/imaging/cosj100020+015344/wavebands//{data.fits,info.json}` for `F115W`, `F150W`, `F277W`, `F444W` — all shipped in this repository | + +The figure is the README's "Getting Started" hero: two rows over the four wavebands, the +top row at a 10" field of view with the 4" mask circle overlaid, the bottom row a 2.5" zoom +carrying each band's measured isophote ellipse. Every annotated number — axis ratio, +position angle, rough half-light radius, peak signal-to-noise, the un-subtracted sky +pedestal — is read at runtime from the per-band `info.json` rather than hard-coded, so the +figure cannot drift out of agreement with the dataset it describes. Where those numbers come +from, and which are measured versus cited, is documented in +[the dataset's own README](../../../dataset/imaging/cosj100020+015344/README.md) and in +[`wiki/core/operations/dataset.md`](../../../wiki/core/operations/dataset.md). + +Rebuild it with: + +```bash +python docs/make_readme_figures.py +``` + +No PyAuto\* import is involved — the script reads the FITS files with `astropy.io.fits` and +draws with `matplotlib` on the `Agg` backend — so it runs in a bare environment with only +`numpy`, `astropy` and `matplotlib` installed. `docs/` is outside the +`autoassistant/audit_skill_apis.py` scan set, so execution from a clean checkout is what +validates it. diff --git a/docs/make_readme_figures.py b/docs/make_readme_figures.py new file mode 100644 index 0000000..7ca03d8 --- /dev/null +++ b/docs/make_readme_figures.py @@ -0,0 +1,467 @@ +""" +COSJ100020+015344 README Figure +=============================== + +Builds the hero figure shown at the top of the README's "Getting Started" section, +which introduces the bundled JWST/NIRCam dataset that both starter prompts point at. + +Everything the figure draws comes from the shipped dataset itself — the four +`data.fits` cutouts and their `info.json` measurements under +`dataset/imaging/cosj100020+015344/`. There are no vendored third-party inputs and no +network access, so the figure rebuilds from a fresh clone of this repository alone. + +The layout is two rows over the same four wavebands, so a feature seen in one column's +top panel reappears in its bottom panel: + +- **Top row** — each band at a 9" field of view with the 4" circular mask a fit of this + galaxy would plausibly use, plus the annotations a first modelling session will ask + about: which blob is the target, the faint neighbour that sits *inside* that mask, and + the un-subtracted sky pedestal. +- **Bottom row** — a 2.5" zoom on the same four bands with each band's own measured + isophote ellipse drawn on it. Priming the reader here means the "how elliptical is it, + and does its size change with wavelength?" beat in a modelling session lands as a + recognition rather than a surprise. + +__Contents__ + +- **Paths**: the shipped dataset inputs and the output figure. +- **Framing**: the two fields of view, the mask radius, and the display stretch. +- **Load**: read each band's cutout and its measurements, and locate the galaxy. +- **Plot**: render the two rows, the mask overlay, the isophote ellipses and the labels. +""" + +import json +from pathlib import Path + +import numpy as np +import matplotlib + +matplotlib.use("Agg") + +import matplotlib.patheffects as path_effects +import matplotlib.pyplot as plt +from matplotlib.patches import Circle, Ellipse +from astropy.io import fits + +"""__Paths__ + +Inputs are the bundled dataset's own products: one `data.fits` plus one `info.json` per +waveband. `info.json` is what makes the annotations measurements rather than guesses — +its `axis_ratio`, `position_angle` and `effective_radius_arcsec_rough` are second-moment +measurements of these same cutouts, and its `background_sky_level` is the sigma-clipped +sky the display stretch subtracts. The dataset README documents where each number comes +from. +""" + +DOCS_PATH = Path(__file__).parent +REPO_PATH = DOCS_PATH.parent + +DATASET_PATH = REPO_PATH / "dataset" / "imaging" / "cosj100020+015344" / "wavebands" +OUTPUT_PATH = DOCS_PATH / "images" / "cosj100020+015344_dataset.png" + +BANDS = ["F115W", "F150W", "F277W", "F444W"] + +"""__Framing__ + +Two fields of view, quoted in arcseconds rather than pixels so the short-wave bands +(0.03"/pixel) and long-wave bands (0.06"/pixel) frame the *same* patch of sky despite +their factor-of-two difference in sampling. + +The 10" top-row field contains the 8"-diameter mask circle with an arcsecond of margin on +each side — enough that the in-panel labels clear the circle rather than colliding with +it. The delivered cutouts are 12.5" on a side, so the crop also trims the outer frame +where the short-wave depth steps across a NIRCam detector-gap boundary. The 2.5" +bottom-row field is roughly three times the galaxy's rough half-light radius — enough to +show the isophote shape without the surrounding sky dominating the panel. +""" + +FIELD_ARCSEC_WIDE = 10.0 +FIELD_ARCSEC_ZOOM = 2.5 + +"""The mask radius the README's second starter prompt would plausibly settle on. The +dataset README's §5c recommends a modelling mask of radius <~ 4": that keeps the fit +inside the uniformly-covered part of the short-wave cutouts and excludes the 8.0" +neighbour, while still reaching well beyond the galaxy's ~0.37" half-light radius. It +does *not* exclude the faint source 2.6" from the centre, which is exactly why that +source is annotated below.""" + +MASK_RADIUS_ARCSEC = 4.0 + +NEIGHBOUR_OFFSET_ARCSEC = 2.6 + +"""The display stretch. `arcsinh` brings the faint outer isophotes and the bright core +into one range without saturating either; scaling the argument by each band's own sky RMS +(rather than a fixed surface brightness) makes the four panels comparable even though +their absolute brightness differs by a factor of several. The sky pedestal is subtracted +for *display only* — the shipped `data.fits` still carries it, which is the point of the +annotation on the F444W panel.""" + +STRETCH_IN_SKY_RMS = 12.0 + +"""__Load__ + +Each band contributes its cutout, its measurements, and a centre. The centre is +recomputed here as a flux-weighted centroid inside a central 2" box rather than assumed +to be the middle pixel: the reduction centred the cutout on the catalogue position, but +the measured centroid sits a fraction of an arcsecond away from the exact centre, and the +mask circle and isophote ellipse should both sit on the galaxy rather than on the grid. +""" + +CENTROID_BOX_ARCSEC = 2.0 + + +def load_band(band): + """The band's sky-subtracted cutout, its `info.json`, and the galaxy's pixel centre.""" + + directory = DATASET_PATH / band + + info = json.loads((directory / "info.json").read_text()) + data = fits.getdata(directory / "data.fits").astype(float) + + data = data - info["background_sky_level"] + + half = int(round(0.5 * CENTROID_BOX_ARCSEC / info["pixel_scale"])) + + centre_y, centre_x = (dimension // 2 for dimension in data.shape) + + box = data[ + centre_y - half : centre_y + half + 1, centre_x - half : centre_x + half + 1 + ] + + """Weighting by the positive flux only keeps sky noise in the corners of the box from + dragging the centroid; the box is small enough that the galaxy dominates regardless.""" + + weights = np.clip(box, 0.0, None) + + rows, columns = np.mgrid[0 : box.shape[0], 0 : box.shape[1]] + + offset_y = float((weights * rows).sum() / weights.sum()) - half + offset_x = float((weights * columns).sum() / weights.sum()) - half + + return { + "info": info, + "data": data, + "centre": (centre_y + offset_y, centre_x + offset_x), + "pixel_scale": info["pixel_scale"], + } + + +bands = {band: load_band(band) for band in BANDS} + +"""__Plot__ + +Four columns (one per waveband) by two rows (wide field, then zoom). Every panel is +square and crops to a field quoted in arcseconds, so the columns stay registered to each +other and the rows stay registered within a column. + +Panels are labelled *inside* their own frame rather than with axis titles, which lets the +grid butt right up against the figure edges — no bands of white space between rows — and +leaves room for the labels to be set large enough to stay readable at the width the README +displays the figure at. `origin="lower"` keeps the FITS row order, which for these +`rotation = 0` mosaics puts north up and east left in every panel. +""" + +"""The panels abut with no gap: any spacing shows through as a white line, since the +figure background is what sits behind the axes. Panels are square, so the figure height is +derived rather than guessed — four panel widths fill the figure width, and two panel +widths give the height.""" + +FIGURE_WIDTH = 14.0 + +PANEL_WIDTH = FIGURE_WIDTH / 4.0 + +figure, axes = plt.subplots(2, 4, figsize=(FIGURE_WIDTH, 2.0 * PANEL_WIDTH)) + +"""Amber for the panel titles reads clearly against the magma colour map and stays +distinct from the two annotation colours already in play — white for the arrows and scale +bar, cyan for the mask and the isophote ellipses. The dark stroke keeps every label +legible where it happens to fall across the bright core.""" + +LABEL_COLOR = "#ffc400" +MASK_COLOR = "#22d3ee" +LABEL_STROKE = [path_effects.withStroke(linewidth=3.0, foreground="black")] + +ARROW_STYLE = dict(arrowstyle="-|>", color="white", lw=1.7, shrinkA=0, shrinkB=3) + + +def crop(band, field_arcsec): + """The band's cutout trimmed to a square field centred on the galaxy, and the pixel + coordinates the galaxy sits at within that crop.""" + + entry = bands[band] + + half = int(round(0.5 * field_arcsec / entry["pixel_scale"])) + + centre_y, centre_x = (int(round(value)) for value in entry["centre"]) + + data = entry["data"][ + centre_y - half : centre_y + half + 1, centre_x - half : centre_x + half + 1 + ] + + """The galaxy's sub-pixel position is preserved relative to the crop so the overlays + land on the galaxy rather than on the nearest whole pixel.""" + + offset_y = entry["centre"][0] - centre_y + offset_x = entry["centre"][1] - centre_x + + return data, (half + offset_y, half + offset_x) + + +def render(ax, band, field_arcsec): + """Draw one band's cropped, arcsinh-stretched image and return its centre in pixels.""" + + entry = bands[band] + + data, centre = crop(band, field_arcsec) + + stretch = STRETCH_IN_SKY_RMS * entry["info"]["background_sky_rms"] + + ax.imshow( + np.arcsinh(data / stretch), + origin="lower", + cmap="magma", + interpolation="bicubic", + ) + + return centre + + +def panel_label(ax, text, color=LABEL_COLOR, fontsize=15): + ax.text( + 0.035, + 0.965, + text, + transform=ax.transAxes, + color=color, + fontsize=fontsize, + fontweight="bold", + ha="left", + va="top", + path_effects=LABEL_STROKE, + ) + + +def footer_label(ax, text, color="white", fontsize=11.5): + ax.text( + 0.035, + 0.035, + text, + transform=ax.transAxes, + color=color, + fontsize=fontsize, + fontweight="bold", + ha="left", + va="bottom", + path_effects=LABEL_STROKE, + ) + + +"""The wide row. Each panel carries its band, its pixel scale and the 4" mask; the four +annotations are spread across the row rather than repeated, so no single panel is +cluttered and the reader collects the whole brief by reading left to right.""" + +for column, band in enumerate(BANDS): + + ax = axes[0, column] + + entry = bands[band] + centre = render(ax, band, FIELD_ARCSEC_WIDE) + + panel_label(ax, f'JWST {band}\n{entry["pixel_scale"]}"/pix') + + ax.add_patch( + Circle( + (centre[1], centre[0]), + radius=MASK_RADIUS_ARCSEC / entry["pixel_scale"], + fill=False, + color=MASK_COLOR, + lw=1.6, + ls="--", + ) + ) + +"""Panel 1 names the target and puts a scale bar on the row. The arrow comes in from the +south-west, away from the annotations on the other panels.""" + +ax = axes[0, 0] +entry = bands["F115W"] +centre_y, centre_x = crop("F115W", FIELD_ARCSEC_WIDE)[1] +span = crop("F115W", FIELD_ARCSEC_WIDE)[0].shape[0] + +ax.annotate( + "COSJ100020+015344\nearly type, z = 0.3422", + xy=(centre_x - 0.35 / entry["pixel_scale"], centre_y - 0.35 / entry["pixel_scale"]), + xytext=(0.10 * span, 0.30 * span), + color="white", + fontsize=12, + fontweight="bold", + ha="left", + va="top", + path_effects=LABEL_STROKE, + arrowprops=ARROW_STYLE, +) + +scale_bar_pixels = 1.0 / entry["pixel_scale"] + +ax.plot( + [0.06 * span, 0.06 * span + scale_bar_pixels], + [0.09 * span, 0.09 * span], + color="white", + lw=2.6, + solid_capstyle="butt", +) +ax.text( + 0.06 * span + scale_bar_pixels / 2.0, + 0.115 * span, + '1"', + color="white", + fontsize=12, + fontweight="bold", + ha="center", + path_effects=LABEL_STROKE, +) + +"""Panel 2 labels the mask, and panel 3 the faint neighbour that sits inside it. The +neighbour is the contaminant question a real-data session has to settle: at 2.6" from the +centre it falls well within any mask wide enough to reach the galaxy's outer isophotes, so +it has to be masked or modelled rather than ignored. Its position angle on the sky is +taken from the data — the brightest pixel in an annulus at that radius.""" + +"""The mask label is anchored in axes fractions at the foot of the panel, below the +circle's lowest arc, so it cannot spill into the neighbouring panel however the crop is +framed.""" + +axes[0, 1].text( + 0.5, + 0.025, + f'{MASK_RADIUS_ARCSEC}" mask — your choice to make', + transform=axes[0, 1].transAxes, + color=MASK_COLOR, + fontsize=11.5, + fontweight="bold", + ha="center", + va="bottom", + path_effects=LABEL_STROKE, +) + + +def brightest_in_annulus(band, radius_arcsec, tolerance_arcsec=0.5): + """Pixel offset (dy, dx) of the brightest pixel in an annulus about the galaxy — used + to point the neighbour arrow at the neighbour actually present in the data.""" + + entry = bands[band] + data, centre = crop(band, FIELD_ARCSEC_WIDE) + + rows, columns = np.mgrid[0 : data.shape[0], 0 : data.shape[1]] + + radius = ( + np.hypot(rows - centre[0], columns - centre[1]) * entry["pixel_scale"] + ) + + annulus = np.abs(radius - radius_arcsec) < tolerance_arcsec + + masked = np.where(annulus, data, -np.inf) + + index = np.unravel_index(np.argmax(masked), masked.shape) + + return index[0] - centre[0], index[1] - centre[1] + + +ax = axes[0, 2] +centre_y, centre_x = crop("F277W", FIELD_ARCSEC_WIDE)[1] + +offset_y, offset_x = brightest_in_annulus("F277W", NEIGHBOUR_OFFSET_ARCSEC) + +"""The arrow tip is in data coordinates (wherever the neighbour actually is) but the label +is anchored in axes fractions at the foot of the panel, so the text stays inside the frame +regardless of which way the neighbour lies. Anchoring it below the source also keeps the +arrow over open sky rather than across the galaxy it is drawing attention away from.""" + +ax.annotate( + f'faint neighbour {NEIGHBOUR_OFFSET_ARCSEC}"\ninside the mask', + xy=(centre_x + offset_x, centre_y + offset_y), + xycoords="data", + xytext=(0.72 if offset_x > 0 else 0.28, 0.035), + textcoords="axes fraction", + color="white", + fontsize=11.5, + fontweight="bold", + ha="center", + va="bottom", + path_effects=LABEL_STROKE, + arrowprops=ARROW_STYLE, +) + +"""Panel 4 carries the sky caveat. It is the one property of this dataset that will +silently bias a light-profile fit without ever looking like an error: the pedestal is real +JWST sky that `calwebb_image3` matched between exposures but never subtracted, and a +Sersic fit that ignores it absorbs it into the profile wings.""" + +"""Kept to two short lines: at this panel width a longer line would run off the frame. The +API for actually freeing the pedestal is in the README and the dataset's own README.""" + +footer_label( + axes[0, 3], + f'sky NOT subtracted: {bands["F444W"]["info"]["background_sky_level"]:.2f} MJy/sr\n' + "model it, don't ignore it", +) + +"""The zoom row. Each panel gets its own band's measured isophote ellipse, drawn at twice +the rough half-light radius so it traces the outer isophotes rather than the core. The +rough radius steps from ~0.29" short-ward to ~0.37" long-ward — a genuine wavelength +dependence mixed with the changing PSF width, and the thing the multi-band starter prompt +asks the assistant to explain.""" + +for column, band in enumerate(BANDS): + + ax = axes[1, column] + + entry = bands[band] + info = entry["info"] + + centre = render(ax, band, FIELD_ARCSEC_ZOOM) + + semi_major = 2.0 * info["effective_radius_arcsec_rough"] / entry["pixel_scale"] + + ax.add_patch( + Ellipse( + (centre[1], centre[0]), + width=2.0 * semi_major, + height=2.0 * semi_major * info["axis_ratio"], + angle=info["position_angle"], + fill=False, + color=MASK_COLOR, + lw=1.6, + ls="--", + ) + ) + + panel_label(ax, f"{band} zoom", fontsize=14) + + footer_label( + ax, + f'q = {info["axis_ratio"]:.2f} PA = {info["position_angle"]:.0f}°\n' + f'R½ ~ {info["effective_radius_arcsec_rough"]:.2f}" S/N = {info["peak_snr"]:.0f}', + color=MASK_COLOR, + ) + +for ax in axes.flatten(): + ax.set_xticks([]) + ax.set_yticks([]) + for spine in ax.spines.values(): + spine.set_visible(False) + +"""With every label moved inside its panel there is nothing left to reserve margin for, so +the axes run to the figure edge and butt against each other.""" + +figure.subplots_adjust(left=0.0, right=1.0, top=1.0, bottom=0.0, wspace=0.0, hspace=0.0) + +"""110 dpi gives a ~1540 pixel wide figure — comfortably sharp at the 900 pixel width the +README displays it at, without the noise-dominated outer regions bloating the committed +PNG.""" + +OUTPUT_PATH.parent.mkdir(parents=True, exist_ok=True) + +figure.savefig(OUTPUT_PATH, dpi=110, facecolor="white") + +print(f"Figure written to: {OUTPUT_PATH.resolve()}") diff --git a/llms.txt b/llms.txt index 797ed0e..8ade08e 100644 --- a/llms.txt +++ b/llms.txt @@ -20,7 +20,7 @@ Read order: [AGENTS.md](./AGENTS.md) → [skills/README.md](./skills/README.md) **⚠️ API currency — read before writing any PyAutoGalaxy code.** Older PyAutoGalaxy releases are heavily represented in model training data and their API is **out of date**. Do **not** write PyAutoGalaxy from memory. The single most common stale error: **plotting is functional now.** Use `aplt.subplot_fit_imaging(fit=fit, output_path=..., output_format="png")` and `aplt.subplot_imaging_dataset(dataset=..., output_path=..., output_filename=..., output_format="png")` — note that `output_filename` is accepted by the dataset subplots, `plot_array` and `plot_grid`, but **not** by `subplot_fit_imaging`, which writes a fixed `fit.png` into `output_path` (passing it raises `TypeError`). The old object-oriented plotters — `aplt.FitImagingPlotter`, `aplt.ImagingPlotter`, `aplt.GalaxyPlotter`, `aplt.GalaxiesPlotter`, `aplt.InversionPlotter` — and the `aplt.MatPlot2D` / `aplt.Include2D` / `aplt.Output` configuration objects have all been **removed**; never emit them. The whole plotting surface is module-level functions, and `dir(autogalaxy.plot)` is the authoritative list. If you can't point at a live example or a `dir()` listing for a call, say it's unverified rather than guessing. -**⚠️ This assistant is young.** Sixteen skills exist: the nine-skill core modelling loop (environment, data preparation, simulation, model building, search configuration, running the fit, plotting, results, debugging), two meta, two project-workflow and three maintenance. The Phase-4b **feature** skills (bases and MGE, pixelisations, ellipse fitting, multi-dataset, interferometry, multi-galaxy, search chaining), the literature wiki and the benchmark suite are still being written, phase by phase. [`skills/README.md`](./skills/README.md) and [`PENDING.md`](./PENDING.md) say exactly what is present and what is planned, and every planned item names the `autogalaxy_workspace` script that grounds it. **Answer from those grounding scripts and say that you did** — never describe a pending skill or wiki page as though you had read it. +**⚠️ This assistant is young.** Sixteen skills exist: the nine-skill core modelling loop (environment, data preparation, simulation, model building, search configuration, running the fit, plotting, results, debugging), two meta, two project-workflow and three maintenance. The Phase-4b **feature** skills (bases and MGE, pixelisations, ellipse fitting, multi-dataset, interferometry, multi-galaxy, search chaining), the literature wiki and the rest of the benchmark suite (one of four cards is written) are still being written, phase by phase. [`skills/README.md`](./skills/README.md) and [`PENDING.md`](./PENDING.md) say exactly what is present and what is planned, and every planned item names the `autogalaxy_workspace` script that grounds it. **Answer from those grounding scripts and say that you did** — never describe a pending skill or wiki page as though you had read it. ## Agent instructions (canonical) @@ -30,9 +30,13 @@ Read order: [AGENTS.md](./AGENTS.md) → [skills/README.md](./skills/README.md) - [skills/README.md](./skills/README.md): the skills that exist, then the phased roadmap of those that don't. +## The bundled dataset + +- `dataset/imaging/cosj100020+015344/` — a real four-band JWST/NIRCam cutout (F115W, F150W at 0.03"/pix; F277W, F444W at 0.06"/pix) of a bright early-type galaxy at spectroscopic z = 0.3422, reduced from COSMOS-Web exposures. Both README starter prompts point at it, so you can answer a real question without the user supplying data. Three things to carry honestly, all documented in [its README](./dataset/imaging/cosj100020+015344/README.md): the **sky is not subtracted** (a positive pedestal 4.5–19x the noise — free `background_sky_level` on an `af.Model(ag.DatasetModel)` or subtract the measured value), the shipped **PSF is a model (STPSF) kernel** and is the dominant systematic, and a **faint neighbour sits 2.6" from the centre**, inside any useful mask. The layout and the `info.json` schema are in [wiki/core/operations/dataset.md](./wiki/core/operations/dataset.md). In chat you cannot plot it yourself — ask the user to, and to confirm the mask extent and contaminants, before composing any fit. + ## Reference -- [wiki/core/index.md](./wiki/core/index.md): curated PyAuto\* reference. The `stack/`, `api/`, `concepts/`, `operations/` and `external/` sections are live; the index itself names the pages still missing (dataset layout, HPC). +- [wiki/core/index.md](./wiki/core/index.md): curated PyAuto\* reference. The `stack/`, `api/`, `concepts/`, `operations/` and `external/` sections are live; the index itself names the pages still missing (the two HPC pages). - [PENDING.md](./PENDING.md): the authoritative ledger of every unwritten skill and wiki page, with its grounding script and target phase. ## Runnable examples & tutorials (elsewhere) diff --git a/skills/ag_prepare_imaging_data.md b/skills/ag_prepare_imaging_data.md index f1d4fcc..697a684 100644 --- a/skills/ag_prepare_imaging_data.md +++ b/skills/ag_prepare_imaging_data.md @@ -430,10 +430,10 @@ unphysical; it is the standard rescue when a search will not converge. Grounded with a click-to-select GUI at `autogalaxy_workspace:scripts/imaging/data_preparation/gui/light_centre.py`. -The on-disk layout these files belong to is not yet a wiki page — until it is, -`autogalaxy_workspace:scripts/imaging/data_preparation/start_here.py` is the ground truth -(the workspace simulates its imaging datasets on first run rather than committing them; -[`../PENDING.md`](../PENDING.md) tracks the page). +The on-disk layout these files belong to is documented in +[`../wiki/core/operations/dataset.md`](../wiki/core/operations/dataset.md), grounded on the +bundled dataset and `autogalaxy_workspace:scripts/imaging/data_preparation/start_here.py` +(the workspace simulates its imaging datasets on first run rather than committing them). ## Combine — where this hands off diff --git a/wiki/core/index.md b/wiki/core/index.md index c6afd52..79d396d 100644 --- a/wiki/core/index.md +++ b/wiki/core/index.md @@ -6,9 +6,9 @@ sources: - wiki/core/ - skills/ - PENDING.md - pinned_commit: ed72fabb33e14a9a701a4d280e8775dd3a20e98c + pinned_commit: db52604f13305cb8a251fb3bb08bb5cc0ab84a55 last_updated: 2026-08-01 -content_sha256: f3ce9c7260b61228d57003c2630d5cbc80ee9eab6606a9fd54cd56ce113bf95b +content_sha256: dceace8177c3d85b8dcf6b1b7abbed457432ece4f7356bd9531d846877641b5c --- # Core wiki — PyAuto\* reference @@ -24,7 +24,7 @@ Five sections, each with a different job: | [`stack/`](#stack) | Which library owns this, and what does it depend on? | 5 | | [`api/`](#api) | Which one exists, and when would I pick it? | 9 | | [`concepts/`](#concepts) | What is this, physically and statistically, and why? | 15 | -| [`operations/`](#operations) | How do I install, configure and run it? | 2 | +| [`operations/`](#operations) | How do I install, configure and run it? | 3 | | [`external/`](#external-resources) | Where do I send the user to read more? | 5 | Every link on this page resolves to a file that exists. Anything not listed here is not @@ -103,6 +103,10 @@ The physics and the inference behind the API. ## Operations +- [Dataset layout and `info.json`](./operations/dataset.md) — the `wavebands//` on-disk + convention, every `info.json` field and where it comes from, how to load one waveband, the + bundled dataset's sky and PSF caveats, and the workspace auto-simulation pattern for data + that is not bundled. - [Installation](./operations/installation.md) — the pip route and its extras, the editable-clone route, version floors and caps, `activate.sh`, and how to prove the install works. @@ -127,10 +131,6 @@ Routing into the three resources outside this repo, by audience. Listed honestly rather than linked. `../../PENDING.md` names the grounding script for each. -- **`operations/dataset.md`** — on-disk dataset layout and `info.json`. Arrives with the - bundled dataset (Phase 2); until then ground data handling in - `autogalaxy_workspace:scripts/imaging/data_preparation/start_here.py` (the workspace - simulates its imaging datasets on first run rather than committing them). - **`operations/hpc.md`** and **`operations/hpc_infrastructure.md`** — cluster concepts, and the `hpc/` templates and `sync` CLI. Both wait on Phase 6 actually shipping `hpc/`; the interface contract a pipeline must preserve is documented meanwhile in diff --git a/wiki/core/operations/dataset.md b/wiki/core/operations/dataset.md new file mode 100644 index 0000000..d0352c9 --- /dev/null +++ b/wiki/core/operations/dataset.md @@ -0,0 +1,240 @@ +--- +title: Dataset layout and info.json +sources: + - project: PyAutoArray + paths: + - autoarray/dataset/imaging/dataset.py + - autoarray/structures/arrays/uniform_2d.py + - autoarray/operators/convolver.py + - autoarray/dataset/dataset_model.py + - autoarray/util/dataset_util.py + pinned_commit: 59b0f198fc7bdf9c91e5a8f734dad796fcc55656 + - project: autogalaxy_workspace + paths: + - scripts/imaging/start_here.py + - scripts/imaging/data_preparation/start_here.py + pinned_commit: d6db2643b9f2cd418efc9473f560dc2a2d459c73 + - project: autogalaxy_assistant + paths: + - dataset/imaging/cosj100020+015344/README.md + - dataset/imaging/cosj100020+015344/info.json + - dataset/imaging/cosj100020+015344/reduction_manifest.json + pinned_commit: 9100260c56c5a5c8eed4e370190ae9c5b94109af +last_updated: 2026-08-01 +content_sha256: b84a33bbbab1269544755a2648e5337bc4b8962ec1b6bf9a523fbd84c13a0a00 +--- + +# Dataset layout and info.json + +How imaging datasets are organised on disk in this repository, what the `info.json` beside +each one actually contains, and how to load a single waveband. There is one bundled dataset +today — the JWST/NIRCam cutout of **COSJ100020+015344** — and this page describes the +convention it establishes rather than a general schema invented ahead of use. + +Two things are deliberately *not* on this page. Data **preparation** (masking, centring, +converting your own `.fits` files into this layout) is a procedure, and belongs to +[`ag_prepare_imaging_data`](../../../skills/ag_prepare_imaging_data.md). The bundled +dataset's **provenance** — which MAST exposures, which pipeline, which catalogue the redshift +comes from — is in the dataset's own README, which is the authority for every number quoted +here. + +## Directory layout + +``` +dataset/ +└── imaging/ # grouped by data type + └── cosj100020+015344/ # one galaxy + ├── README.md # provenance: exposures, pipeline, caveats, citations + ├── info.json # dataset-level summary + every band nested under `per_band` + ├── reduction_manifest.json # per-band reduction and PSF record + └── wavebands/ + ├── F115W/ # 419 x 419 @ 0.03"/pix + │ ├── data.fits + │ ├── noise_map.fits + │ ├── psf.fits # 21 x 21, unit sum + │ └── info.json # this band's measurements + ├── F150W/ # 419 x 419 @ 0.03"/pix + ├── F277W/ # 209 x 209 @ 0.06"/pix + └── F444W/ # 209 x 209 @ 0.06"/pix +``` + +Two conventions are load-bearing: + +- **`wavebands//` is one dataset directory per band.** A multi-band set is *not* one + directory with suffixed filenames. Each band carries its own `data.fits`, + `noise_map.fits`, `psf.fits` and `info.json`, because each band has its own pixel scale, + its own PSF and its own measured shape. Single-band datasets may sit directly under + `dataset/imaging//` with no `wavebands/` level. +- **The pixel scale differs between bands.** The NIRCam short-wave bands are sampled at + 0.03"/pixel and the long-wave bands at 0.06"/pixel. Reading `pixel_scale` from the band's + own `info.json` rather than hard-coding one value for the dataset is not optional — get it + wrong and every angular quantity the fit reports is wrong by a factor of two, silently. The + four cutouts cover the same ~12.5" field despite the different array shapes. + +## Loading one waveband + +`ag.Imaging.from_fits` (`PyAutoArray:autoarray/dataset/imaging/dataset.py`) takes the three +FITS paths plus the pixel scale: + +```python +import json +from pathlib import Path + +import autogalaxy as ag + +dataset_path = Path("dataset") / "imaging" / "cosj100020+015344" / "wavebands" / "F277W" + +info = json.loads((dataset_path / "info.json").read_text()) + +dataset = ag.Imaging.from_fits( + data_path=dataset_path / "data.fits", + noise_map_path=dataset_path / "noise_map.fits", + psf_path=dataset_path / "psf.fits", + pixel_scales=info["pixel_scale"], +) +``` + +**`dataset.psf` is a `Convolver`, not an array.** This catches people out: the attribute +returns the convolution operator (`PyAutoArray:autoarray/operators/convolver.py`), and the +kernel array itself is `dataset.psf.kernel` — an `Array2D`, so `dataset.psf.kernel.native` +is the 21x21 numpy view and `.shape_native` its shape. Writing `dataset.psf.native` from +memory raises `AttributeError`. + +Data are in **MJy/sr**, NIRCam's native surface-brightness unit as `calwebb_image3` writes +it, not converted to counts. Nothing in PyAutoGalaxy cares which flux unit you use as long as +the data, noise map and any `intensity` prior agree, but a prior written for counts will be +wildly mis-scaled here. + +## `info.json` + +Two levels, with the same field names at each. The **per-band** `info.json` under +`wavebands//` describes that band. The **dataset-level** `info.json` at the dataset root +promotes one band's values to the top (`reference_band`, F277W for this dataset — highest +signal-to-noise and uniform coverage), adds a cross-band scatter, and nests all four bands +under `per_band`. + +### Every field, and where it comes from + +All of these are **measured from the delivered cutouts** by the reduction run's own +measurement script. That is the point of the schema: nothing in `info.json` is copied from a +catalogue and nothing is estimated by eye. + +| Field | Level | Meaning | +|---|---|---| +| `pixel_scale` | both | Arcsec/pixel for this band — 0.03 short-wave, 0.06 long-wave | +| `centre_ra`, `centre_dec` | both | Flux-weighted centroid of the central source (degrees, J2000), through the cutout WCS | +| `axis_ratio` | both | `semiminor/semimajor` from second-order central moments, measured over the source's own segment so a neighbour cannot pull the shape | +| `position_angle` | both | Major-axis angle from the same moments, degrees counter-clockwise from +x, wrapped to [0, 180) | +| `effective_radius_arcsec_rough` | both | **Rough and biased low.** Half-light radius of an elliptical curve of growth truncated at 3". *Not* a Sersic `effective_radius` — a prior scale, not a measurement to reproduce | +| `moment_semimajor_sigma_arcsec` | per-band | Second-moment semimajor sigma from the same measurement | +| `peak_snr`, `peak_value` | per-band | Max of `data / noise_map`, and of the background-subtracted data, over the source segment | +| `background_sky_level` | per-band (and `background_sky_level_per_band` at dataset level) | Sigma-clipped median outside every detected segment — the **un-subtracted** sky, see below | +| `background_sky_rms` | per-band | Standard deviation of that same sky estimate | +| `segment_area_pixels` | per-band | Pixel count of the detection segment the measurements were made over | +| `curve_of_growth_max_arcsec` | per-band | The 3" truncation radius the rough half-light radius was measured inside | +| `data_name`, `reference_band` | dataset | The dataset's name, and which band's values are promoted to the top level | +| `band_to_band_scatter` | dataset | Standard deviation of `axis_ratio` and `effective_radius_arcsec_rough` across the four bands — **the honest error bar on those two numbers** | +| `measurement_note` | dataset | Prose restating the `_rough` caveat in the file itself | +| `per_band` | dataset | The four per-band blocks, keyed by band name | + +### What is *not* in `info.json`, on purpose + +- **No redshift.** COSJ100020+015344's z = 0.3422 is a published spectroscopic value, not + something measured from these pixels, so it is cited in the dataset README instead of + sitting in a file that otherwise contains only measurements. Read it from the README (or + ask the user) and pass it to `ag.Galaxy(redshift=...)` explicitly. +- **No `mask_radius`.** There is no default mask hiding in the metadata. The mask extent is a + science decision the assistant must settle with the user on every real-data fit — a mask + that truncates the outer isophotes biases the effective radius and Sersic index directly. + See the real-data gate in [`../../../AGENTS.md`](../../../AGENTS.md). +- **No magnitudes, stellar mass or physical size.** The surface-brightness-to-AB conversion, + aperture correction and Galactic-extinction treatment needed to publish those honestly were + not carried out. The dataset README cites COSMOS2020 photometry for anyone who needs them. +- **No catalogue identifiers.** They are in the README, with their VizieR table references. + +## Two caveats that change how you fit this data + +Both are properties of *this* dataset rather than of the layout, but an agent that loads the +data without reading them will produce a biased fit that looks fine. + +**The sky is not subtracted.** `calwebb_image3`'s skymatch step matches the exposures' +backgrounds to each other but does not remove them, so `data.fits` carries the real JWST sky +as a positive pedestal — between 4.5x and 19x the median noise depending on the band. A +light-profile fit that ignores it absorbs the pedestal into the profile wings and returns an +inflated effective radius and Sersic index. Either subtract `background_sky_level` first, or +model it: `background_sky_level` is a constructor parameter of `ag.DatasetModel` +(`PyAutoArray:autoarray/dataset/dataset_model.py`), so freeing it means assigning a prior on an +`af.Model` of that class — + +```python +import autofit as af + +dataset_model = af.Model(ag.DatasetModel) +dataset_model.background_sky_level = af.UniformPrior(lower_limit=0.0, upper_limit=0.4) + +model = af.Collection(galaxies=..., dataset_model=dataset_model) +``` + +Note that `af.Model(ag.DatasetModel)` starts with **no free parameters** — every constructor +argument holds its default until you assign a prior to it, so omitting the assignment above +silently fixes the sky at 0.0 rather than fitting it. `background_sky_level` is not a class +attribute either; `ag.DatasetModel.background_sky_level` raises `AttributeError`. See +[`../concepts/sky_background_and_operated_profiles.md`](../concepts/sky_background_and_operated_profiles.md). + +**The PSF is a model PSF.** `psf.fits` is an STPSF model kernel composed per detector and +position, not an empirical star-stacked kernel — the pipeline's empirical PSF builder fails on +this field, because its star selection is dominated by compact galaxies rather than stars. The +shipped kernels are physically sane and correctly ordered with wavelength, but come out +consistently *sharper* than an empirical reference, most so in the most undersampled band +(F115W). Expect a fitted size to compensate slightly large, and **treat the PSF as the +dominant systematic in any fit to this dataset**. The measured comparison, and why the +empirical kernels were rejected, are in the dataset README; `reduction_manifest.json` records +both the shipped kernel and the rejected one per band. + +## Datasets that are *not* bundled + +Most `autogalaxy_workspace` examples do not ship their data — `dataset/.gitignore` there +ignores the whole tree (`*`, with only itself un-ignored), and each example **simulates its +dataset on first run** instead. The pattern, from +`autogalaxy_workspace:scripts/imaging/start_here.py`, is a guard plus a subprocess call to the +matching simulator: + +```python +if ag.util.dataset.should_simulate(str(dataset_path)): + subprocess.run( + [sys.executable, "scripts/imaging/features/extra_galaxies/simulator.py"], + check=True, + ) +``` + +`ag.util.dataset.should_simulate` (`PyAutoArray:autoarray/util/dataset_util.py`) is an +**existence check on the directory** — it returns `not Path(dataset_path).exists()`, with one +extra behaviour: when `PYAUTO_SMALL_DATASETS=1` is set it *deletes* an existing dataset first, +so the simulator regenerates it at the reduced resolution and the FITS shapes cannot disagree +with the capped mask and grid. Two consequences worth knowing: + +- Because the check is existence-only, a **partially written or stale** dataset directory is + treated as present and will not be regenerated. If a simulated dataset looks wrong, delete + the directory rather than re-running the script. +- Never point this pattern at a bundled dataset. The guard would see + `dataset/imaging/cosj100020+015344/` exists and skip — but under `PYAUTO_SMALL_DATASETS=1` + it would **delete real observational data** that no simulator can regenerate. + +For a user's own data, the job is to get it into the layout at the top of this page and then +load it exactly as above; the conversion procedure, including how to build a noise map and PSF +when the archive does not hand you one, belongs to +[`ag_prepare_imaging_data`](../../../skills/ag_prepare_imaging_data.md), grounded in +`autogalaxy_workspace:scripts/imaging/data_preparation/start_here.py`. + +## See also + +- [`api/datasets.md`](../api/datasets.md) — the `Imaging` and `Interferometer` objects, their + settings, masking and over-sampling. +- [`concepts/grids_and_masks.md`](../concepts/grids_and_masks.md) — why the mask extent and + the over-sampling scheme change your answer. +- [`concepts/multi_wavelength.md`](../concepts/multi_wavelength.md) — fitting the four bands + jointly rather than one at a time. +- [`operations/sandbox.md`](./sandbox.md) — `PYAUTO_SMALL_DATASETS`, `PYAUTO_TEST_MODE` and + the writable-cache environment variables. +- The bundled dataset's own `README.md` — the authority for its provenance, its measurements + and its caveats.