perf: let per-hash and per-conversation telemetry reads seek instead of sorting

[FrameAutomata]

Closes #335. Stacked on #334 — based on that branch so the diff here is one commit. Retarget to main once #334 merges.

idx_exceptions_project_hash stopped at the hash column, so nothing covered (project_id, exception_hash, recorded_at). This widens it. The old index stays a leading prefix, so every query that used it stays covered.

The issue's diagnosis was wrong in both directions

#335 predicted the paginated per-hash read would sort the group's whole history, and asked to "confirm the USE TEMP B-TREE FOR ORDER BY disappears". There is no temp B-tree in that plan to begin with — and more importantly, the old index has a pathology in two different plans, and which one you get flips on whether ANALYZE has ever run. Measuring only one stats state would have justified the change for the wrong reason and missed the worse half.

• Without stats, SQLite walks idx_exceptions_project_recorded in recorded_at order and re-filters the hash per row. Cost scales inversely with how noisy an issue is: an issue with fewer occurrences than the page holds can never satisfy the LIMIT, so it walks the entire project. That is the common case for a just-appeared issue — exactly when someone opens the page.
• With stats, it switches to the hash index and sorts. That fixes the two reads above and instead makes the notification dispatch path 2.35s per fired exception.

The widened index is the only shape that is fast in both states.

Measurements

File-backed 2M-row, 500-hash, 30-day database (8.1GB, WAL), median of 3-5 runs. The multi-second figures are I/O-bound and move with page-cache warmth, so they are given as ranges across runs; the sub-millisecond ones were stable.

	old	old + ANALYZE	new	new + ANALYZE
issues list (24h)	4.9-24.8s	305ms	361ms	340ms
issue page, 5 occurrences	16.4-33.7s	63µs	44µs	54µs
issue page, 252k occurrences	335-479µs	268µs	323µs	167µs
notify: latest occurrence	64µs	2.35s	37µs	29µs
notify: post-archive count	15µs	409ms	23µs	23µs
no-data poll: MAX(recorded_at)	11µs	28µs	21µs	12µs
CountByHour (control)	40.5ms	39.6ms	40.2ms	43.2ms

PRAGMA optimize (retention/sqlite.go:90) is a no-op on a fresh connection and did not create sqlite_stat1, so the no-stats column is the likelier production state — but both are reachable, which is the point.

Two query sites outside the repository file turned out to be in scope and were not in the issue: the new-error evaluator's latest-occurrence and post-archive-count reads (notifications/evaluator_event_sqlite.go). Both are on the ingest-driven dispatch path.

The no-data poll's MAX(recorded_at) is the query with the most to lose — SQLite's min/max optimisation answers it in one seek only because recorded_at sits directly after project_id in idx_exceptions_project_recorded, which this does not touch. EXPLAIN confirms it keeps that index in all four configurations.

Write cost

Below measurement noise. Three alternating runs: 19.8-22.6k rows/s on the old index, 20.1-23.2k on the widened one — overlapping ranges. This replaces an index rather than adding one, so it does not repeat the ~2x insert penalty the genuinely-new tasks index cost in #334. The entry is ~29% wider (a 30-char timestamp), about +60MB here. Building it took ~10s.

The DROP-then-CREATE ordering is safe because the migration version is recorded only after every statement succeeds (migrations.go:73) and boot panics on migration failure (cmd/run.go:121) — a killed index build re-runs cleanly rather than serving traffic without the index. That property is load-bearing and currently implicit.

Guard test

telemetry_group_index_test.go grew a time-column field, so it now covers check_results, log_records (which uses timestamp, not recorded_at) and metric_points alongside the four recorded_at tables. Those three were already correctly indexed but unguarded — metric_points is the one with actual drift history (0006 did this same drop-and-widen). Verified the test fails without 0021 and passes with it.

Not done here

• idx_ai_traces_project_conversation stops short of recorded_at: FindByConversationId can never fill its LIMIT #336 — idx_ai_traces_project_conversation has the identical defect, and FindByConversationId has a 1000-row LIMIT it can never fill with an optional time bound. Unmeasured, so it gets its own issue rather than an unmeasured index here — the same split that produced exception_stack_traces per-hash reads sort the whole group: idx_exceptions_project_hash stops short of recorded_at #335 from Telemetry per-group queries scan a group's whole history: indexes stop short of recorded_at #323.
• Index-building migrations run silently at boot with no startupProbe to survive them #337 — this and perf: cover recorded_at in the per-group telemetry indexes #334 together add four index builds to boot. Migrations log nothing and there is no startupProbe in the Helm chart, so a large-database upgrade can be killed mid-build by the liveness probe and crash-loop.
• No ClickHouse or DuckDB change. CH already has a bloom_filter skip index on exception_hash (ch/0014), which is the idiomatic equivalent — its ORDER BY (project_id, recorded_at, exception_hash) alone would not prune here, since the per-hash reads leave recorded_at unbounded. DuckDB is columnar and carries no secondary indexes.
• The main DB's vestigial exception_stack_traces (sqlite/0002_telemetry_tables.up.sql) is untouched. Every repository query site uses db.TelemetryDB, a physically separate file, so that copy never receives rows; perf: cover recorded_at in the per-group telemetry indexes #334 left its endpoints twin alone for the same reason.

[dusanstanojeviccs]

We should only have PRs against main

[FrameAutomata]

Retargeted to main. The base branch was perf/323-telemetry-group-time-indexes, whose PR (#334) has since merged — rebasing dropped both of its commits as already upstream, leaving one commit here.

The diff is now just the migration (0021_index_exceptions_hash_time.up.sql) and the guard test it extends. go test ./app/migrations/ -run TestTelemetryGroupIndexes passes on the rebased branch.

Covers #335, which you noted can be safely added.

---

Generated by Claude Code

[FrameAutomata]

Extended to also cover #336, which you approved on the same terms ("created at could be part of the index... we should still do this").

Both are the same defect and the same guard test, so splitting them would have produced two PRs against main editing the same file — the merge-order problem you raised, in a smaller form.

0022_index_ai_traces_conversation_time.up.sql widens idx_ai_traces_project_conversation to (project_id, conversation_id, recorded_at). FindByConversationId orders by recorded_at with LIMIT 1000, and a LIMIT cannot short-circuit a sort — so the old index made it read and sort the entire conversation to return the first page, however long the conversation is.

Measured on a 150k-turn conversation in a 350k-row table, both ANALYZEd:

	plan	time
before	SEARCH (project_id=? AND conversation_id=?) + USE TEMP B-TREE FOR ORDER BY	31.6ms
after	SEARCH (project_id=? AND conversation_id=? AND recorded_at>? AND recorded_at<?)	0.5ms

The old plan scales with conversation length; the new one is flat.

One thing worth recording since it shapes how much this matters: without ANALYZE, and only when the request carries a date range, SQLite still prefers idx_ai_traces_project_recorded and re-filters conversation_id per row. That is the second bad plan the test comment describes, and it is why this is a guard test rather than a benchmark — which of the two plans you get depends on statistics that may never have been gathered. With statistics, or with no date range, the new index wins outright.

Both are agreed low-severity for the reason you gave — DuckDB is the self-hosted telemetry backend, and DuckDB is columnar with no secondary indexes, so this is SQLite-only by construction. It costs one migration file.

---

Generated by Claude Code