Configurable alerting: per-type toggles, quiet hours, error-state alerts, per-session cost budgets

[xerhab]

Context

heartbeatAlerts() + the offline sweep in agent-hub/server.js push five edge-triggered ntfy alert types (offline/recovered, restart loop, daily host cost ≥ COST_ALERT_USD, question waiting, PR created, turn finished). Configuration is minimal: one global cost threshold env, and everything else is always-on.

Problem / gaps

1. No per-type control: "turn finished" is high-value during an active work session and noise overnight; there's no way to mute a type without unsetting NTFY_URL and losing offline alerts too.
2. No quiet hours: every alert fires 24/7 at fixed priority.
3. Sessions entering error state never alert. _set_error() on the agent puts the message in the payload and the card shows it — but nothing pushes. A session that failed to spawn/resume can sit in error for days unnoticed unless you look at the dashboard. This is arguably the most important missing alert.
4. Cost granularity: the threshold is per-host-per-day only. A single runaway session can't trip a budget of its own, and there's no fleet-wide monthly view.

Proposal

• Error-state alert (quick win, independent): in heartbeatAlerts(), fire on a session's transition into status === "error" (edge-tracked in alerts.sessions[id] like the others), tags warning, including errorMsg.
• Config: a small ALERTS config (env-JSON or a /data/alerts.json beside the state file) with per-type enable + priority overrides, quiet-hours window (suppress default-priority during it, let high/urgent through), and thresholds: costPerHostDay (today's), costPerSessionDay, optionally costFleetMonth.
• Budget enforcement (opt-in escalation): a per-session daily budget with action: "alert" | "kill" — the hub already knows per-session usage.today.cost on every beat and can queue the existing kill command. Killing preserves branch/usage/resumability, so it's a safe circuit breaker for a session stuck in a loop.
• Settings UI can start as documentation of the env/JSON (matching the repo's inline-env convention) — a dashboard settings panel is a possible follow-up but not required for the value.

Touchpoints

• agent-hub/server.js — heartbeatAlerts(), notify() (priority/quiet-hours gate), config load
• Sibling DockerOps compose — new env documented in this repo's README/CLAUDE.md