diff --git a/README.md b/README.md index 2355582..ecc0921 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,8 @@ between dashboards, and the service installer's `--name` does it all per charger in one command. See [More than one Wall Connector](https://github.com/zebraengine/wallmonitor/blob/main/docs/running.md#more-than-one-wall-connector). All history lands in a single `wallmonitor.db` SQLite file — back up that -one file and you have everything. For every option, and for running it as a +one file and you have everything (`wallmonitor --backup DIR` writes a +verified, compressed snapshot wherever you point it, while running). For every option, and for running it as a systemd service on an always-on box, see [Running wallmonitor](https://github.com/zebraengine/wallmonitor/blob/main/docs/running.md). ## At a glance diff --git a/deploy/install-service.sh b/deploy/install-service.sh index 090d0b6..ae0b9ea 100755 --- a/deploy/install-service.sh +++ b/deploy/install-service.sh @@ -7,6 +7,11 @@ # sudo ./install-service.sh --host 192.168.1.50 --bind 0.0.0.0 --port 8480 # sudo ./install-service.sh --uninstall # +# Optional daily backup (nothing is scheduled unless you ask): a verified, +# compressed snapshot into a directory of your choosing — local, a NAS +# mount, or a folder a sync client watches — every night at 03:30: +# sudo ./install-service.sh --host 192.168.1.50 --backup-dir /srv/backups/wallmonitor +# # More than one Wall Connector: give each its own --name. That yields a # separate unit (wallmonitor-), a separate database (.db) and # the label shown in that instance's UI and notifications; link the @@ -38,6 +43,7 @@ UNINSTALL="0" NAME="" LABEL="" PEERS=() +BACKUP_DIR="" usage() { grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit "${1:-0}"; } @@ -55,6 +61,7 @@ while [[ $# -gt 0 ]]; do --name) NAME="$2"; shift 2 ;; --label) LABEL="$2"; shift 2 ;; --peer) PEERS+=("$2"); shift 2 ;; + --backup-dir) BACKUP_DIR="$2"; shift 2 ;; --uninstall) UNINSTALL="1"; shift ;; -h|--help) usage ;; *) echo "unknown argument: $1" >&2; usage 1 ;; @@ -82,6 +89,8 @@ if [[ "$(id -u)" -ne 0 ]]; then fi if [[ "$UNINSTALL" == "1" ]]; then + systemctl disable --now "${SERVICE_NAME}-backup.timer" 2>/dev/null || true + rm -f "/etc/systemd/system/${SERVICE_NAME}-backup.timer" "/etc/systemd/system/${SERVICE_NAME}-backup.service" systemctl disable --now "$SERVICE_NAME" 2>/dev/null || true rm -f "$UNIT_PATH" systemctl daemon-reload @@ -147,8 +156,42 @@ fi echo "WantedBy=multi-user.target" } > "$UNIT_PATH" +# Opt-in daily backup: nothing is scheduled unless --backup-dir is given. +BACKUP_UNIT="${SERVICE_NAME}-backup" +if [[ -n "$BACKUP_DIR" ]]; then + ENV_LINES="" + [[ -n "$DB_PATH" ]] && ENV_LINES="Environment=WM_DB=${DB_PATH}" + cat > "/etc/systemd/system/${BACKUP_UNIT}.service" < "/etc/systemd/system/${BACKUP_UNIT}.timer" < ${BACKUP_DIR} (systemctl list-timers ${BACKUP_UNIT}.timer)" +fi echo "" echo "installed and started. Useful commands:" diff --git a/docs/recording.md b/docs/recording.md index d5fde57..9f249f3 100644 --- a/docs/recording.md +++ b/docs/recording.md @@ -51,7 +51,8 @@ the recording pipeline in the order data flows through it. - **Storage** is a single SQLite file (`wallmonitor.db`, WAL mode) next to where you run the app, or wherever `--db` points. Back up that one file and - you have your complete history. There is no retention limit; expect very + you have your complete history — `--backup DIR` does it safely while the + monitor runs (see [Backup](running.md#backup)). There is no retention limit; expect very roughly 10–25 MB/day depending on how often a vehicle is attached. The charger itself keeps no history — the monitor's database *is* the history, starting from the first time it runs. diff --git a/docs/running.md b/docs/running.md index 6226c5c..02075a1 100644 --- a/docs/running.md +++ b/docs/running.md @@ -47,6 +47,9 @@ is the charger's microcontroller. | `--peer LABEL=URL` | `WM_PEERS` | — | Link another instance from the header switcher; repeatable (env: comma-separated) | | `--retain-raw-days` | `WM_RETAIN_RAW_DAYS` | `0` (off) | Trim raw JSON from samples older than N days (min 7); columns stay forever | | `--compact` | — | — | One-shot VACUUM to reclaim trimmed space; run with the service stopped | +| `--backup DIR` | `WM_BACKUP_DIR` | — | One-shot verified, compressed snapshot into DIR, rotate, exit; safe while running | +| `--backup-compress` | `WM_BACKUP_COMPRESS` | `gzip` | `gzip` (~12×, seconds), `xz` (~22×, minutes), or `none` | +| `--backup-keep` | `WM_BACKUP_KEEP` | `7/4/12` | Keep newest per day / week / month for that many; `0/0/0` keeps all | | `--port` | `WM_PORT` | `8480` | Web UI port | | `--bind` | `WM_BIND` | `127.0.0.1` | Web UI bind address | | `--db` | `WM_DB` | `wallmonitor.db` | SQLite path | @@ -165,6 +168,45 @@ service and run: uv run python -m wallmonitor --compact --db /path/to/wallmonitor.db ``` +## Backup + +The database is the history — the charger keeps none — and the +degradation watch judges every new charge against it. One command makes a +copy worth having: + +```bash +uv run python -m wallmonitor --backup /path/to/backups --db /path/to/wallmonitor.db +``` + +It takes a consistent snapshot with SQLite's online backup API (safe while +the service runs — about 4 s for a 1.4 GB database), runs +`PRAGMA integrity_check` on the copy and refuses to continue if it isn't +`ok`, compresses it (gzip by default: a 1.4 GB database becomes ~110 MB in +about 15 s; `--backup-compress xz` halves that again in a few minutes), and +places it as `wallmonitor--.db.gz` with an atomic +rename, so a sync client never uploads a half-written file. Then it rotates: +the newest snapshot per day for 7 days, per week for 4 weeks, per month for +12 months (`--backup-keep D/W/M`). Only files matching its own naming +pattern are ever deleted — anything else in the folder is left alone. + +**Where the folder is, is yours to decide**, and wallmonitor never talks to +anything beyond it: a second disk; a NAS mount; a folder that iCloud Drive, +Sync, Syncthing or Dropbox already watches; or simply a directory another +machine pulls from — `rsync -a user@box:/path/to/backups/ ~/wallmonitor-backups/` +from the *other* machine keeps credentials off the always-on box. + +Nothing is scheduled by default. To run it daily under the service's user, +install with `--backup-dir DIR` (below), or add your own cron/timer entry. + +**Restore:** stop the service, then + +```bash +gunzip -c wallmonitor--.db.gz > /path/to/wallmonitor.db +rm -f /path/to/wallmonitor.db-wal /path/to/wallmonitor.db-shm # stale WAL would corrupt the restored file +``` + +and start the service. The file is plain SQLite; nothing else is needed. + ## Run as a service (Ubuntu / systemd) For an always-on box, `deploy/install-service.sh` installs wallmonitor as a diff --git a/tests/test_backup.py b/tests/test_backup.py new file mode 100644 index 0000000..e54474e --- /dev/null +++ b/tests/test_backup.py @@ -0,0 +1,134 @@ +"""Backup: verified snapshot, compression, atomic placement, rotation.""" + +import datetime as dt +import gzip +import os +import sqlite3 +import time + +import pytest + +from wallmonitor import backup +from wallmonitor.config import parse_args +from wallmonitor.db import Database + + +@pytest.fixture +def db(tmp_path): + (tmp_path / "src").mkdir() + database = Database(str(tmp_path / "src" / "wallmonitor.db")) + yield database + database.close() + + +def _fill(db, n=200): + base = time.time() - 3600 + for i in range(n): + db.insert_vitals(base + i * 10, {"handle_temp_c": 30.0 + i * 0.01, "vehicle_current_a": 48.0, + "pilot_high_v": 8.9, "prox_v": 1.5}, 1, 11000.0) + + +def _stamp(y, m, d, h=3): + return dt.datetime(y, m, d, h, 30, tzinfo=dt.timezone.utc).timestamp() + + +def test_backup_snapshot_is_verified_compressed_and_readable(db, tmp_path): + _fill(db) + db.set_setting("device_serial", "PGT-123 456") # odd chars get sanitized in the name + dest = tmp_path / "dest" + result = backup.run_backup(db.path, str(dest), "gzip", now=_stamp(2026, 8, 27)) + assert result.integrity == "ok" + assert os.path.basename(result.path) == "wallmonitor-PGT-123_456-20260827T033000Z.db.gz" + assert result.output_bytes < result.snapshot_bytes + assert result.deleted == () + # Nothing but the finished file in the destination — no temp, no snapshot. + assert sorted(os.listdir(dest)) == [os.path.basename(result.path)] + # And no snapshot left beside the source either. + assert sorted(os.listdir(os.path.dirname(db.path))) == ["wallmonitor.db", "wallmonitor.db-shm", "wallmonitor.db-wal"] \ + or all(not name.startswith(".wallmonitor-snapshot") for name in os.listdir(os.path.dirname(db.path))) + # The compressed copy is a complete, consistent database. + restored = tmp_path / "restored.db" + with gzip.open(result.path, "rb") as fin, open(restored, "wb") as fout: + fout.write(fin.read()) + conn = sqlite3.connect(str(restored)) + assert conn.execute("PRAGMA integrity_check").fetchone()[0] == "ok" + assert conn.execute("SELECT COUNT(*) FROM vitals_samples").fetchone()[0] == 200 + assert conn.execute("SELECT value FROM settings WHERE key='device_serial'").fetchone()[0] == "PGT-123 456" + conn.close() + # The live database kept working throughout (the snapshot was read-only). + db.insert_vitals(time.time(), {"handle_temp_c": 31.0}, 1, 0.0) + + +def test_backup_unpinned_serial_and_no_compression(db, tmp_path): + _fill(db, 5) + result = backup.run_backup(db.path, str(tmp_path / "dest"), "none", now=_stamp(2026, 1, 2)) + assert os.path.basename(result.path) == "wallmonitor-unpinned-20260102T033000Z.db" + assert sqlite3.connect(result.path).execute("PRAGMA integrity_check").fetchone()[0] == "ok" + + +def test_backup_xz(db, tmp_path): + _fill(db, 5) + result = backup.run_backup(db.path, str(tmp_path / "dest"), "xz", now=_stamp(2026, 1, 2)) + assert result.path.endswith(".db.xz") and result.output_bytes < result.snapshot_bytes + + +def test_rotation_keeps_newest_per_bucket_and_leaves_foreign_files(): + # 40 consecutive daily snapshots plus strays: rotation is judged from the + # timestamp in the name (sync clients rewrite mtimes) and never touches + # files it didn't write. + names = [] + for i in range(40): + when = dt.datetime(2026, 8, 27, 3, 30, tzinfo=dt.timezone.utc) - dt.timedelta(days=i) + names.append(f"wallmonitor-SER1-{when:%Y%m%dT%H%M%SZ}.db.gz") + names += ["wallmonitor-SER1-20260827T010000Z.db.gz", # same day, earlier: not the newest of that day + "wallmonitor-OTHER-20260827T033000Z.db.gz", # a different charger's file + "notes.txt", "wallmonitor.db", "wallmonitor-SER1-garbage.db.gz"] + doomed = set(backup.rotate(names, "SER1", backup.Keep(7, 4, 12))) + kept = set(names) - doomed + # Foreign / other-serial / malformed names are never deleted. + for foreign in ("wallmonitor-OTHER-20260827T033000Z.db.gz", "notes.txt", "wallmonitor.db", + "wallmonitor-SER1-garbage.db.gz"): + assert foreign in kept + # The 7 newest days survive; the earlier same-day copy does not. + for i in range(7): + when = dt.datetime(2026, 8, 27, 3, 30, tzinfo=dt.timezone.utc) - dt.timedelta(days=i) + assert f"wallmonitor-SER1-{when:%Y%m%dT%H%M%SZ}.db.gz" in kept + assert "wallmonitor-SER1-20260827T010000Z.db.gz" in doomed + # Weekly tier: newest per ISO week for 4 weeks; monthly: newest per month. + survivors = sorted(n for n in kept if n.startswith("wallmonitor-SER1-2026")) + weeks = {dt.datetime.strptime(n[17:32], "%Y%m%dT%H%M%S").isocalendar()[:2] for n in survivors} + months = {n[17:23] for n in survivors} + assert len(weeks) >= 4 and {"202607", "202608"} <= months + # 40 dailies collapse to well under half. + assert 7 <= len(survivors) <= 14 + # 0/0/0 keeps everything. + assert backup.rotate(names, "SER1", backup.Keep(0, 0, 0)) == [] + + +def test_backup_rotation_deletes_on_disk(db, tmp_path): + _fill(db, 5) + db.set_setting("device_serial", "SER1") + dest = tmp_path / "dest" + dest.mkdir() + (dest / "keep-me.txt").write_text("not yours") + for d in range(1, 12): + backup.run_backup(db.path, str(dest), "none", backup.Keep(3, 0, 0), now=_stamp(2026, 8, d)) + names = sorted(os.listdir(dest)) + assert "keep-me.txt" in names + snaps = [n for n in names if n.startswith("wallmonitor-SER1-")] + assert [n[17:25] for n in snaps] == ["20260809", "20260810", "20260811"] + + +def test_backup_refuses_corrupt_source(tmp_path): + bad = tmp_path / "bad.db" + bad.write_bytes(b"SQLite format 3\x00" + b"\x00" * 4000) + with pytest.raises(Exception): + backup.run_backup(str(bad), str(tmp_path / "dest")) + assert not (tmp_path / "dest").exists() or os.listdir(tmp_path / "dest") == [] + + +def test_backup_flag_needs_no_host(tmp_path): + cfg = parse_args(["--backup", str(tmp_path / "b"), "--db", str(tmp_path / "x.db")]) + assert cfg.backup_dir == str(tmp_path / "b") and cfg.backup_compress == "gzip" + with pytest.raises(SystemExit): + parse_args(["--backup", "x", "--backup-keep", "7/4"]) diff --git a/wallmonitor/__main__.py b/wallmonitor/__main__.py index d9994bb..2b96b3c 100644 --- a/wallmonitor/__main__.py +++ b/wallmonitor/__main__.py @@ -65,6 +65,16 @@ async def run(argv: list[str] | None = None) -> None: cfg = parse_args(argv) if cfg.discover is not None: raise SystemExit(await run_discovery(cfg.discover, split_phase_hint=cfg.split_phase)) + if cfg.backup_dir: + from .backup import Keep, run_backup + + result = run_backup(cfg.db_path, cfg.backup_dir, cfg.backup_compress, Keep.parse(cfg.backup_keep)) + print(f"{result.path}: {result.snapshot_bytes / 1e6:.1f} MB snapshot (integrity {result.integrity}, " + f"{result.snapshot_s:.1f} s) -> {result.output_bytes / 1e6:.1f} MB " + f"{cfg.backup_compress} ({result.compress_s:.1f} s)") + if result.deleted: + print(f"rotated out {len(result.deleted)}: " + ", ".join(result.deleted)) + raise SystemExit(0) if cfg.compact: db = Database(cfg.db_path) try: diff --git a/wallmonitor/backup.py b/wallmonitor/backup.py new file mode 100644 index 0000000..a6e9f77 --- /dev/null +++ b/wallmonitor/backup.py @@ -0,0 +1,222 @@ +"""One-shot database backup: consistent snapshot, verified, compressed, +rotated — into a directory the user chooses. + +Where that directory *is* — a local path, a mounted NAS, a folder a sync +client watches (iCloud Drive, Sync, Syncthing, Dropbox), or somewhere a +second machine pulls from with rsync — is deliberately none of this +module's business. wallmonitor never talks to a cloud; it writes a file. + +Order of operations, and why: + +1. Snapshot with SQLite's online backup API into a temp file *next to the + source* (never in the destination: a sync-watched folder would upload a + transient uncompressed copy). Consistent under WAL while the service + keeps polling — measured at ~4 s for a 1.4 GB database. +2. ``PRAGMA integrity_check`` on the snapshot before compressing. A backup + that is corrupt but looks fine is the worst outcome; better to fail loud. +3. Compress (gzip by default: ~12x, seconds; xz: ~22x, minutes) to a temp + name in the destination, fsync, atomic rename — a sync client never sees + a half-written file. +4. Rotate: keep the newest per day / per ISO week / per month for the + configured counts, judged by the date in the filename (sync clients + rewrite mtimes). Only files matching this module's own naming pattern + are ever deleted; anything else in the folder is left alone. +""" + +from __future__ import annotations + +import datetime as _dt +import gzip +import lzma +import os +import re +import shutil +import sqlite3 +import tempfile +import time +from dataclasses import dataclass + +COMPRESSIONS = ("gzip", "xz", "none") +_SUFFIX = {"gzip": ".db.gz", "xz": ".db.xz", "none": ".db"} +_NAME_RE = re.compile( + r"^wallmonitor-(?P[A-Za-z0-9_-]+)-(?P\d{8})T(?P