Prometheus /metrics endpoint on the hub for Grafana integration

[xerhab]

Context

The home lab already runs a Grafana/ntfy observability stack (the hub's ntfy alerting references grafana.yaml in DockerOps). The hub holds rich fleet state — host online/offline, per-session status/working/waiting, token counts, API-equivalent cost, container memory — but exposes it only as the UI-shaped /api/agents JSON behind basic auth.

Problem

Fleet metrics can't join the existing dashboards/alerting in Grafana: no long-term charts of cost vs. time next to other lab metrics, no Grafana alert rules on agent data, no recording of host memory trends. Anything wanting the data must scrape and reshape the ad-hoc JSON.

Proposal

• Add GET /metrics to agent-hub/server.js rendering Prometheus text format from the in-memory agents map — pure string formatting, no dependency, consistent with the stdlib-only constraint. Candidate series:
  • agenthub_host_online{host,device} 0/1, agenthub_host_last_seen_seconds, agenthub_host_memory_used_bytes / _limit_bytes
  • agenthub_sessions{host,status} gauge
  • agenthub_session_working{host,session,repo} 0/1, agenthub_session_waiting_question 0/1
  • agenthub_session_tokens_total{host,session,repo,kind="input|output|cacheWrite|cacheRead"} and agenthub_session_cost_dollars_total — from usage.totals, which are monotonic per session, so they work as counters
  • agenthub_tunnel_connected{host} 0/1 (from controlChannels)
• Auth: Prometheus can send basic auth, so the existing user gate can protect it; alternatively a dedicated METRICS_TOKEN bearer to avoid putting the UI password in scrape config.
• Cardinality note: per-session labels churn as sessions come and go — acceptable at home-lab scale; a METRICS_PER_SESSION=0 toggle can restrict to host-level series if it ever matters.

Touchpoints

• agent-hub/server.js — /metrics route + formatter
• DockerOps (sibling) — scrape config; documented here