Skip to content

fix: drift watch — gate truncated fits, buzz the phone only for confirmed drift - #21

Merged
zebraengine merged 2 commits into
mainfrom
fix/drift-fit-quality
Aug 27, 2026
Merged

fix: drift watch — gate truncated fits, buzz the phone only for confirmed drift#21
zebraengine merged 2 commits into
mainfrom
fix/drift-fit-quality

Conversation

@zebraengine

@zebraengine zebraengine commented Aug 27, 2026

Copy link
Copy Markdown
Owner

Problem

A "Heat rise climbing vs baseline" alert reached the phone on 08-12 at high priority: n=8+3, Δ +4.65 °C, 95% CI [−0.8, +10.1], confident: false. Replaying it from the live DB showed two things wrong at once:

  1. Truncated fits. Two of the eight baseline fits came from steady-current windows of 11.6 and 14.3 min against an 11 min τ. Ending before the plateau shows, the exponential can explain the samples with a lower rise and a faster τ (7.0–7.2 min) and still pass every gate with RMSE ≈ 0.2. They fit ~8 °C under the rest of history and dragged the baseline median from ~32.7 to 31.1. Across all 27 fits corr(τ, rise) = +0.74 — the fingerprint of that trade-off.
  2. A lead interrupts like a conviction. Post-anchor session-to-session scatter is ~3.4 °C (MAD-sigma 3.0). With a 3-session recent median the DRIFT_WARN_C = 2.5 tripwire sits at ~1.07σ, i.e. ~14% chance of drifting: true per look with no real change. detect_drift already reports this honestly through confident, but the ntfy push went at high either way.

Code touched

  • wallmonitor/thermal.py — new MIN_SPAN_TAU = 1.8; fit_sessions drops a segment whose steady window spans less than 1.8× max(install median τ, DEFAULT_TAU_MIN) — not the fit's own τ (a truncated segment can't vouch for itself), and floored so a fresh install's very first fit is judged against a sane prior rather than nothing. 1.8τ ≈ 83% of the rise observed.
  • wallmonitor/thermal.py — the steady-prefix window is no longer a fixed 30 min (tuned to an 11 min τ, and unclearable for a τ ≈ 20 install). It is max(30 min, PREFIX_SPAN_TAU=2.5 × τ) from the install's τ prior; if a segment then fits slower than the prior that sized its window, the window is widened to the fitted τ and refit once, so a slow-τ install — or the first fit on an empty one — can bootstrap.
  • wallmonitor/poller.pyrecheck_thermal_drift sends kind thermal_drift (priority high) when confident, else thermal_drift_lead (priority default, tags wrench,mag). Alert row and thermal_drift event unchanged.
  • tests/test_wallmonitor.pytest_thermal_fit_rejects_window_short_against_tau (1.0τ rejected, exactly-1.8τ accepted); test_thermal_fit_first_fit_judged_against_default_tau (12 min first charge on an empty install rejected); test_thermal_fit_slow_tau_install_still_fits (τ = 20 min install, four hour-long charges, all four fit); test_thermal_drift_confirmed_notifies_high_priority / test_thermal_drift_lead_notifies_default_priority; derate-midway test's full-rate phase 20 → 25 min to clear the gate at τ=12.
  • docs/thermal-model.md, docs/notifications.md — the gate and the priority split.

Risk

  • The floor at DEFAULT_TAU_MIN (12 min) means an install with a genuinely fast τ (say 6 min) needs ≥ 21.6 min of steady current per fit instead of ~11 min; short top-offs there stop teaching the model. Accepted: a fast-τ install has nothing to lose from waiting for the plateau, while a fresh install with no floor had its first fit unguarded.
  • DEFAULT_TAU_MIN = 12 and the other Tier-3 priors (idle-offset calibration, DRIFT_WARN_C) are still values from one install. This PR stops them from starving a different install of fits; making the drift threshold self-calibrating and flagging when a fresh install's fitted τ lands far from the default are follow-ups, not in scope here.
  • The widen-and-refit pass fetches a second, larger window for segments that fit slower than the prior — one extra vitals_range call per such segment, bounded at two passes. Negligible next to the 40-session refit cost.
  • Leads now arrive quietly. If the user relies on the phone for drift and has push set to only show high+, an unconfirmed drift is dashboard-only until it confirms. That is the intended behaviour and documented.
  • JSON-format webhook consumers filtering on kind == "thermal_drift" will not see leads; they now come as thermal_drift_lead.

Verification

  • uv run pytest -q: 111 passed (106 before + 5 new).
  • Replayed the new fit_sessions read-only against the production DB: 27 fits → 23. Dropped sid 30 (11.6 min window), sid 32's both segments (19.2 and 14.3 min), sid 47 (15.8 min) — every window under 1.8 × 12 min. Segments whose τ fitted slower than the prior (e.g. sid 25, τ 15.2) were refit on a widened window; their rise moved by < 0.7 °C. Model params: τ 11.0 → 11.25 min, rise_ref 34.8 → 35.3 °C.
  • Replayed the 08-12 verdict with the gated fits: baseline 31.1 → 32.7, Δ 4.65 → 3.72, still drifting, still not confident — so the gate alone would not have silenced it, but under this PR it goes out as a default-priority lead instead of a high-priority buzz. The one low baseline value left (sid 36, 27.9 °C on a 34 °C day) is a separate effect: the amp controller capping hot sessions selects the cooler-running ones into the 48 A bucket. Not addressed here.
  • Live verdict after the gate: Δ +0.74 °C, CI [−3.3, +4.8], no drift.

Deploy: restart the service (fit_sessions runs in-process; the 6 h fit cache is in memory). No migration; fits are recomputed from raw vitals on every read, so the chart and drift verdict pick up the gate immediately.

Fernando Gonzalez and others added 2 commits August 26, 2026 20:28
…rmed drift

Two fixes for the degradation watch, both traced from a real "Heat rise
climbing" ntfy alert (n=8+3, Δ 4.65 °C, CI straddling zero).

Fit gate: a steady-current window that ends before the plateau shows
cannot separate rise from tau — the fitter trades a lower rise for a
faster tau and passes every other gate with a fine RMSE. On the live
install two such windows (11.6 and 14.3 min against an 11 min tau) fit
8 °C under the rest and sat in that alert's baseline, dragging its median
down. fit_sessions now requires the window to span >= 1.8x the install's
median tau (judged against the install's tau, not the fit's own biased
one). The derate-midway test's full-rate phase grows from 20 to 25 min so
it clears the gate at tau=12.

Notification priority: with ~3.4 °C session-to-session scatter and a
3-session recent median, DRIFT_WARN_C sits near one sigma, so an
unconfirmed verdict is a lead, not a conviction — detect_drift already
says so via `confident`. The alert and event are unchanged; the ntfy push
now goes at default priority (kind thermal_drift_lead) unless the
confidence interval clears zero, which keeps high priority.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… by tau

The identifiability gate compared a window against the install's median
tau, which on a fresh install is the first fit's own tau — exactly the
quantity a truncated window biases low. Floor it at DEFAULT_TAU_MIN so
the first charge a new user records is held to a sane prior.

The steady-prefix window was a fixed 30 min, tuned to an 11 min tau. An
install with a 20 min tau could never span 1.8 tau inside it and would get
no fits at all. The window now scales with the tau estimate (2.5 tau,
30 min floor), and when a segment fits slower than the prior that sized
its window, the window is widened to the fitted tau and refit once — so
a slow-tau install, or the first fit on a fresh one, can bootstrap.

Tests: a 12 min first charge on an empty install is rejected; a tau=20
install with hour-long charges fits all four sessions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant