perf(programs): resolve the winning runs once, and stop sorting rows nobody reads in order (ADR-087)

[Taleef7]

/programs on the pilot was not slow, it was failing. Reproduced from the numbers below, not from
the screenshot: GET /api/programs/overview answers 503 statement_timeout at 30 s on a cold
process, and the page renders "Failed to load program data" over "No active measures. Create and
release a measure to begin." — a database timeout shown to a quality lead as an empty catalog.

Measured first (live Maui, 20,000 patients, 17:59–18:20Z, outside the nightly window)

request	cold	warm
GET /api/programs/overview	503 at 30 s (twice)	3.2–3.7 s
GET /api/programs/overview?include=detail&granularity=month	—	59.8 s
GET /api/programs/sites — asked on every page load	17.2 s	2.5 s
GET /api/programs/:id/trend?granularity=month	—	3.0–4.8 s × 6
GET /api/programs/:id/top-drivers	—	0.42–0.72 s
GET /api/cases?status=open&limit=25 (ADR-084's fast path)	—	0.50 s
GET /api/exports/outcomes?runId=<winner>	—	120,000 rows / 27.8 MB in 40.9 s

The 59.8 s is the finding, and it is not the total. It was measured with every downstream memo
already warm — so the whole minute was the winners walk, paid thirteen times by one page load.
programSites isolates it: warm, its only remaining work is latestPopulationWinners, and it takes
2.5 s to return five strings.

Three costs, one page

1 · The winners walk is not memoized. listLatestPopulationRuns is two questions in one method.
WHICH runs qualify is one indexed statement over runs (bounded LIMIT, milliseconds). WHICH of them
holds a row for each measure is an EXISTS probe per (run, measure), and outcomes has no
(run_id, measure_id) index — so the pair costs 2.5–4 s, and every read model resolved its own.

The probe is now memoized under the candidate run list itself, per store instance. The cheap
statement runs every call and IS the key. That identity is exact rather than a TTL: a probe's answer
depends only on which runs are in the list and which rows they hold, a terminal population run's rows
are immutable — the fact every RunKeyedMemo and the roster cell cache already rest on — and a run
enters the list only once terminal.

Two writes can change a probe's answer without moving the key, and both invalidate it. An outcome
write can add the first row of a measure to a run already in the list — the import-driven
finalize does exactly that — and compaction can delete a non-winner's last row for a measure. So
recordOutcome, recordOutcomes and compactOlderThan drop the memo, on both stores. A winner
cannot be affected by compaction (its rows are keep-set rows by construction, ADR-073), but the memo
does not know that, and a cache that has to reason about which entries are still safe is a cache
nobody can audit.

2 · aggregateOfficialRun sorted one measure's whole evidence eleven times — this is the 503.
listOutcomes orders by (evaluated_at, id) and no index serves it, so an ordered read of one measure
of a whole-population run sorts the measure's 20,000 rows, evidence_json and all. The aggregate paid
that once for a one-row provenance probe and once per LIMIT/OFFSET page, each page re-running the
same filter and the same sort to skip further into it. Six measures, on the cold path.

Now ONE unordered statement. listOutcomes gained order: "none" — opt-in, for a caller that folds
its rows; a paged read keeps its ordering whatever the caller asks, because paging an unordered
relation may repeat or skip rows. The aggregate also reports producedOfficialEvidence, so
officialMeasureRate reads the rows once instead of probing and then aggregating.

That the overview's other reads are not the culprit is why the export row is in the table: 120,000
rows with evidence come back in one statement without timing out, and every other read the overview
makes is smaller than that.

3 · Every memo is in-process, so a deploy guarantees a cold dashboard. warmReadModels has existed
since #547 and runs after each population run — which covers the nightly and misses every restart, the
one moment the caches are certainly empty. It now runs at boot too, off the request path, twice if
the first attempt fails, never once shutdown has begun, and best-effort exactly as the post-run warm
is. Two greppable log lines, documented in DEPLOY.md.

What this does NOT fix, named rather than implied

CREATE INDEX … ON outcomes (run_id, measure_id) is almost certainly the largest single win left,
and it is owner DDL. It would take the probes from 2.5–4 s to index lookups and give the evidence
read a plan that does not sort. Additive, reversible, no data migration — the same class as the three
OWNER-APPROVED DDL blocks already in schema-pg.ts. Recommended with the measurements in ADR-087 and
DEPLOY.md, and deliberately not written: schema is the owner's. What ships here removes repeated
work; the index would make each unit of work cheap. They are not substitutes.

The overview still derives its status buckets from 120,000 rows in JavaScript, where
GROUP BY measure_id, status, out_of_population over the winners returns about 90. ADR-084's own
pattern applies, but the site, tenant and profile predicates read the in-memory directory and have no
SQL form — so it is a fast-path-plus-fallback with a conformance test. Deferred on purpose: worth
measuring this change before shipping two performance stories in one breath.

Honest about the method

The endpoint timings are measured on the live stack. WHICH statement the server cancelled is
inferred from the read shapes: this host cannot run a pilot-scale Postgres (2.0 GB free of 15.2,
Docker down) and the deployment gives no query log, so there is no EXPLAIN ANALYZE behind the
attribution. Recorded as inference in ADR-087 so nobody later cites it as a measurement. The AFTER
numbers are measured on the deploy and written into JOURNAL.md and guide ch. 9 — never predicted.

Verification

backend-ts suite	2,856 tests — 2,832 pass, 23 skip, 1 fail
tsc --noEmit	clean
pnpm test:shards:verify	exhaustive, disjoint, Pg files on the serviced shard
new tests	16 (5 ProbeCache, 5 ceiling-SQL, 2 store-contract × both stores, 3 rate, +1 paging)
mutation check	removing the write-path invalidation fails the new store-contract case

The one failure is corpus-membership.test.ts — this host's .official-content sparse checkout, the
same pre-existing failure #609 carried. Unrelated to these files.

The Pg ceiling self-skips here (no local postgres:16), and both #544 defects passed the SQLite
floor for exactly that reason. So the ceiling's assembled SQL is pinned without a server:
outcome-store-sql.test.ts drives PgOutcomeStore with a recording pool and asserts the emitted
text — that order: "none" drops exactly the ORDER BY and strands no keyword, that binds still line
up, that a paged read keeps its sort, and that a second identical winners call re-reads the candidate
list but not the probe. The store-contract tests remain the real guarantee and run in CI.

---

Review round: eight findings, five of them defects in the above

My own code review of this branch found eight things. Two would have put the timeout back, and one
was a fresh vacuous guard created while fixing three others. Commit ae90ba97.

The memo reached the wrong store. getStores caches its bundle in a WeakMap keyed by the env
OBJECT, and a live container builds two: the schedulerEnv literal in server.ts and the one the host
builds for the worker. Only the pool is module-global, so there are two PgOutcomeStore instances.
Per-instance, the compaction invalidation landed on the instance that does the DELETING and stayed open
on the one that does the READING — and the boot warm filled a cache no request would ever hit, so the
cold ?include=detail this PR exists to fix was essentially unimproved. Now keyed by the database
handle.

officialMeasureRate did not memoize null, and the null now costs a full read. programOverview
runs that loop per request outside its own memo, so a measure with no official evidence (an
all-errored nightly, a run predating a flip, any authored measure) would have re-read its whole
evidence on every dashboard load, forever. The 30-second cliff, reintroduced by the fix for it.

The aggregate lost its memory bound and my note for it was wrong twice. I claimed the unpaged read
was "smaller than the 120,000-row read the overview already makes" and that rows are "folded as they
arrive". listOutcomesWithRun has a lean projection with no evidence_json at all, and
for (const row of await …) awaits the whole parsed array. The bound is back as a projection —
listOutcomeMembershipsForRun returns the official object and the error marker, the few dozen bytes
run-aggregate.ts already said a sum needs. This matters beyond the dashboard: the subject cap gates
the individual and bundle MeasureReport variants only, so on a 20,000-patient run the summary report
and QRDA III are the reachable exports and this is what bounds them. routes/runs.ts's comment
claiming the cap covered them is corrected.

producedOfficialEvidence was decided by an arbitrary row. Unordered, "first evaluated row" is
whatever the planner returns, and the answer is memoized — so one unreadable populationResults
arriving first would have made an official measure read as authored and dropped its rate off the
dashboard, non-deterministically. Any evaluated row now settles it. My reason for keeping the first-row
rule (duplicate alerts) was false: aggregator.add already calls officialMembership on every
non-error row.

The boot warm's retry could not fire, and its success line printed on failure. warmReadModels
swallows every error by design — including the cold-connection failure the retry was keyed on — so
read models warmed at boot was logged whatever happened, and DEPLOY.md points an operator at that
line. It returns a result now. A control that reads as present and cannot fire is the defect class this
project keeps finding, and I added one while removing three.

Three tests were missing and one was vacuous

Removing runs.map((r) => r.id) from the probe key	suite stayed green — and that key is the only thing invalidating a new nightly on the request-serving instance
The paged-ordering guard	compared three rows whose insertion order was their sort order, so it passed either way
Compaction's invalidation	had no test at all

All three are pinned and mutation-checked. The fakes now derive their membership read from their own
fixture (test-support/memberships.ts), so the two readers cannot drift.

Stated in ADR-087 rather than fixed

• A third write can move a probe's answer at a fixed key: finalizeRun flips a run into the
  qualifying set after its rows were written, so a run too old for the 25-run probe budget can enter
  step 2's reckoning without the candidate list moving. Two adjacent statements of one finalize, and it
  takes a backdated rerun to reach.
• Stratum ORDER within a rate is now row-order dependent (a Map's insertion order). Differs only
  where one (run, measure)'s rows carry heterogeneous stratifier sets; every stratum carries its own
  id, so a consumer keying by id is unaffected.
• The export path still pays the old sort once, via runProducedOfficialEvidence, and only when the
  routing flag is off — so on the pilot it short-circuits and this is TWH's authored rosters.

Verification after the review round

backend-ts suite	2,869 tests — 2,845 pass, 23 skip, 1 fail (corpus-membership, pre-existing, this host's sparse checkout)
tsc --noEmit	clean
pnpm test:shards:verify	exhaustive, disjoint, Pg files on the serviced shard
frontend suite / lint	491/491 · clean (2 pre-existing warnings)
mutations killed	6 (probe key, per-instance cache, floor paging, floor projection, and two audit-order cases)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 927e78d04d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Include fallback state in the probe-cache key

When the probe budget cannot satisfy a measure, the cached value also includes results from the fallback query over runs outside this candidate list, but the key contains only the top candidate IDs. If a separate writer adds backdated terminal runs below that budget—for example the seed:trend-history CLI while the server is running—the candidate IDs remain unchanged and the server instance is not cleared by the CLI's store, so trend requests continue returning the old winners indefinitely. Either avoid caching fallback-derived answers or include an identity that changes with the older qualifying rows.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Propagate warm-up failures so the boot retry runs

When a read-model query fails after getStores succeeds, warmReadModels catches the overview failure (and each per-measure failure) and returns normally. This call therefore logs that the models were warmed and exits the loop instead of taking the advertised second attempt, leaving the first /programs request to pay the cold read after exactly the transient query failure this boot logic is intended to recover from. The warmer needs to report failure to this retry layer, or return a success/failure result that is checked here.

Useful? React with 👍 / 👎.