Optional retention: trim old raw JSON, keep every column forever

[zebraengine]

Problem

Nothing is ever deleted, and the database grows ~1 GB/month on a busy install — measured on production: 1.12 GB in 45 days, of which vitals_samples is 94 % and the raw JSON blobs are ~84 % of that (~600 of ~712 B/row). Backups, disk on the always-on box, and every copy-for-analysis workflow degrade monthly. Roadmap item from #1 ("database stays near-flat on long-running installs").

Code touched

wallmonitor/config.py — --retain-raw-days N (WM_RETAIN_RAW_DAYS; default 0 = never trim, preserving the current promise; minimum 7 so a typo can't gut a database) and --compact (one-shot maintenance run: VACUUM, print sizes, exit; exempt from the --host requirement like --discover).

wallmonitor/db.py

• trim_raw(cutoff_ts) blanks raw to '' (the column is NOT NULL) on vitals/wifi/lifetime/ambient samples older than the cutoff, in 10k-row chunks each holding the write lock briefly. A per-table timestamp cursor in settings makes the daily pass scan only rows that newly aged past the cutoff. Vitals are gated on the diagnostics backfill being complete — the backfill's source is the raw blob. forecast_samples (tiny; raw carries fields with no column, e.g. steady_state_se_c history) and version_info are never trimmed.
• vacuum() checkpoints the WAL (TRUNCATE) before and after VACUUM so the reported sizes mean what they say.
• Two hardenings the new tests caught: the pre-backfill diagnostics fallback now CASE-guards json_extract — SQLite does not promise to short-circuit COALESCE past json_extract(''), which is a hard error, not NULL; and the backfill UPDATE skips raw = '' rows.

wallmonitor/__main__.py — the backfill task grows into _maintenance: one-time backfill, then (when retention is enabled) a daily trim pass, with counts logged. --compact handled before any component starts.

wallmonitor/poller.py — retain_raw_days exposed on /api/status for future UI honesty.

Docs — running.md gains a Retention section (what is kept, what re-interpretability costs, why the file plateaus rather than shrinks, how --compact reclaims); recording.md's fidelity claim and the README bullet now reference the policy.

Deliberately not in scope: downsampling old samples to a coarser cadence (phase 2 if ever needed — columns at full cadence are only ~16 % of the growth once raw is trimmed), automated backups, CSV export.

Risk

• Opt-in and conservative by default: 0 keeps today's behavior bit-for-bit. Nothing is deleted even when enabled — rows, columns, sessions, events, alerts, forecasts all survive; only the raw blob of old samples is blanked.
• The real cost is re-interpretability: extracting a field that was never a column (exactly how the diagnostics columns were backfilled in Faster session pages: diagnostics as columns, gzip, rounded JSON #14) becomes impossible for trimmed rows. The docs say this in so many words.
• Interaction with the backfill is ordered by construction (vitals gate) and error-proofed both directions (CASE guard, raw != '').
• --compact is exclusive while it runs; documented to run with the service stopped. The daily trim itself never shrinks the file — freed pages are reused, which is the "near-flat" behavior wanted.

Verification

• 106/106 tests. New tests/test_retention.py: cutoff respected with columns intact and served through both vitals_range branches on trimmed rows; the vitals gate; cursor resumes instead of rescanning; VACUUM reclaims the trimmed bytes (WAL-checkpointed sizing); flag validation (3 rejected, 30 accepted, default off). The first run of these tests caught both hardening bugs above — the COALESCE non-short-circuit would have broken pre-backfill queries on any trimmed database.
• Measured on a full 45-day production snapshot (1,479,693 vitals rows): trim_raw at 14-day retention touched 1,127,172 rows in 8.3 s; --compact shrank the file 1331.7 → 636.5 MB in 6.7 s. Steady state at 14-day retention: raw exists only for the newest two weeks (~250 MB of blobs) plus columns forever (~5 MB/month) — the file plateaus.

Deploy: git pull + restart; enabling retention is a per-install decision (WM_RETAIN_RAW_DAYS in the service unit). Suggested for the reference install: 90 days to start — comfortably beyond the thermal model's 120-day column lookback (which retention doesn't touch anyway) and every past reanalysis window, while still capping growth.

🤖 Generated with Claude Code