Skip to content

Epic: watcher inherits the shared VM — neutral broker node, cross-host bus, cohort decommission #282

Open
Open
Epic: watcher inherits the shared VM — neutral broker node, cross-host bus, cohort decommission#282
Labels
documentationImprovements or additions to documentationenhancementNew feature or request

Description

@gregoryfoster
gregoryfoster
opened on Aug 31, 2026

Watcher's half of the cohort split. Fourth and last of the four migrations opened by notifier#43 — and the only one that is not a move. Watcher stays on this VM; archiver and replicator leave, and the broker leaves with neither of them. What migrates here is watcher's couplings, not its data.

Why this one is shaped differently

notifier#43 moved a service to a new host. Watcher is the host: node watcher / tag:watcher / 100.120.218.69 / watcher.exe.xyz is this box, and it also carries archiver, replicator, the archiver-operated broker, and the shared Postgres 16 cluster. Migrating watcher by subtraction means:

  • no pg_dump, no restore, no cutover outage, no Fernet-key-equivalent gate. notifier's R2 — the risk that actually carried that migration — has no analogue here.
  • the bus becomes the migration. Notifier was one outbound HTTP client with a 10 s timeout. Watcher publishes four streams and consumes two, in both directions, as long-lived in-process loops. Loopback Redis becomes a WireGuard hop.
  • the epic completes only when archiver and replicator have left. Phases below are ordered by that dependency, not by watcher's own readiness.

The one thing inherited wholesale is the closing note of notifier#43: a service split is not a disk-pressure remedy. That VM went 971 M → 1.1 G free and stayed at 95 %. This one is at 1.1 G free / 95 % used today. The measured reclaim is set out in R7 — and as on notifier, it is the venvs, not the databases, that carry it. The retention work in Phase 5 reclaims about as much as both departures combined, which is why it is a phase and not a footnote.

Audit — what actually couples

Watcher's footprint on the shared VM

Asset Location
Units watcher.service (:8000, prod DB), watcher-cleanup.timer/.service (weekly)
Secrets /etc/watcher/.env → DATABASE_URL, WATCHER_MIGRATION_DATABASE_URL, WATCHER_BUS_REDIS_URL, GOOGLE_APPLICATION_CREDENTIALS, GCS_BLOB_CREDENTIALS
Unit-only credential /etc/watcher/notifier.env (0600 root:root, #277/#278)
GCS keys /etc/watcher/co-gcs-blob-reader.json, /etc/watcher/co-pypi-reader.json
Repo secrets .env → 7 × GH_TOKEN_*, TEST_DATABASE_URL, ANTHROPIC_API_KEY, WATCHER_DEV_NOTIFIER_*
Postgres roles watcher (schema owner) + watcher_app (DML-only, #259/#270); DBs watcher (312 MB), watcher_test, watcher_a1..a4_test
Runtime /run/watcher/build-id
Tailnet node watcher, tag:watcher, 100.120.218.69, non-expiring
Sibling dependency ARCHIVER_REPO_PATH (default /home/exedev/archiver) — conftest runs archiver's alembic to build the information schema

312 MB is misleading. procrastinate_events is 183 MB and procrastinate_jobs is 118 MB; every domain table together is ~11 MB (audit_log 1.5 MB, fetch_commands 504 kB, watched_items 96 kB, change_revisions 88 kB). 96 % of the database is job churn with no retention policy anywhere — scripts/cleanup.sh prunes the uv cache and dangling Docker images and does not touch Postgres.

The bus — the coupling that this epic is actually about

Redis runs here today on 127.0.0.1:6379, appendonly no, no requirepass, archiver-operated (archiver#109).

The client is built for loopback and only loopback. src/core/bus.py:147 is the entire construction:

return Redis.from_url(os.environ[BUS_REDIS_URL_ENV])

No socket_timeout, no socket_connect_timeout, no health_check_interval, no retry_on_timeout, no reconnect policy. On loopback those defaults never mattered. Across a WireGuard hop they are the difference between a logged reconnect and a wedged consumer — see R1.

Inbound — who reaches watcher

Outbound

Notifier at http://notifier:9000 / :9001 over the tailnet (already migrated, #280). GCS for the cannobserv wheelhouse and blob reads. Origin fetches are replicator's, not watcher's — co-core owns fetch → extract → fingerprint and watcher no longer fetches at all.

Not coupled

Nothing in src/ mirrors to a sibling (#159, #236). The archiver SDK and its path dependency were removed in #254; the only remaining sibling-repo dependency is ARCHIVER_REPO_PATH for conftest's alembic run, and that is a checkout dependency, not a service dependency — it survives archiver's service leaving, as long as the checkout does.

Decisions

# Decision Rationale
D1 Watcher stays; archiver and replicator move out. Watcher is the largest tenant, holds the Postgres cluster, and is the only one with an operator UI on the proxy. Moving the other two is strictly less work than moving it, and leaves tag:watcher describing what it says.
D2 The broker moves to a neutral node, owned by neither watcher nor archiver. Today's topology has archiver operating a broker that watcher, archiver and replicator all depend on. Leaving it on archiver's new VM would make one participant the mesh's centre and every other service's availability a function of archiver's deploys. A dedicated node costs one more VM and removes that.
D3 Exposure unchanged: 0.0.0.0:8000 behind the exe.dev proxy's login gate. No tailnet-only bind, no nftables restriction. Verified: eth0 is 10.42.0.42 on this VM and notifier's audit reported 10.42.0.42 on that one — 10.42.0.0/16 is a per-VM namespace, not a shared tenant LAN, so the 0.0.0.0 bind exposes nothing to other tenants. The only external path is the proxy, which is login-gated by default. Taking this decision means skipping notifier's D3 and most of R1 — no tailnet_bind.sh in watcher's ExecStart, no boot race in the HTTP path, loopback curl keeps working.
D4 The dashboard stays unauthenticated; the network boundary is the boundary. Recorded as a decision, not an oversight. The consequence is stated plainly in R6: the exe.dev public/private bit is the entire authentication story for the admin UI, and it is one share set-public away from being wrong.
D5 Retention before anything else. Prune procrastinate_events/procrastinate_jobs and give them a standing policy. Originally scoped as a cutover-dump argument; with D1 there is no dump, so the justification is now purely disk — 301 MB of 1.1 G free. Weaker, still real, and it is the only phase here that reclaims anything meaningful.
D6 Node identity unchanged: stays watcher / tag:watcher / 100.120.218.69. notifier#43's hardest-won operational lesson is that tags bind at device registration — retagging needs logout + fresh up, cheap on a new node and not on one peers depend on. D1 means we never have to. As archiver and replicator leave, tag:watcher narrows from "three services" to "watcher" by subtraction, and the existing tag:watcher → tag:notifier:9000,9001 rule becomes precise without being edited.
D7 Broker auth is requirepass at minimum, Redis ACL users preferred. observo-primary is a user-owned node on this tailnet. A tailnet-bound broker with no password is reachable by every node the ACL admits. Per-service Redis ACL users would additionally scope watcher to its six streams. Broker owner's call — recorded here as watcher's requirement.
D8 Retain the archiver checkout at /home/exedev/archiver after archiver's service leaves. Exactly notifier#43's precedent, for the same reasons: ARCHIVER_REPO_PATH needs it for conftest's alembic, and SOCRATICODE_LINKED_PROJECTS resolves through it. A read-only convenience, not a deploy target.

What the split does not change

Stated once so it is not assumed away:

Prerequisites

  • Cohort agreement on D2 — the broker's home is not watcher's to decide alone. Needs archiver's and replicator's owners.
  • exe.dev API token. Not present in this repo's .env; notifier's was scoped new/ls/whoami without rm, which was the right privilege. Note new has no --region flag and set-region was outside that scope — the reason notifier landed in pdx while this VM is lax.
  • SSH access to the new broker node. notifier#43 stalled on exactly this: ssh was not in the token's cmds allow-list and no local key was registered. Decide the admin path before provisioning, not after.
  • A tag:broker auth key — single-tag, Pre-approved, Tagged, non-Ephemeral, generated before first join. notifier#43's F1: a dual-tag key applies its tag set wholesale and cannot be narrowed by --advertise-tags.
  • Tailscale ACL updated before key generation.

Proposed policy addition:

{
  "tagOwners": {
    "tag:broker":     ["autogroup:admin"],
    "tag:archiver":   ["autogroup:admin"],
    "tag:replicator": ["autogroup:admin"]
  },
  "acls": [
    // Every bus participant reaches the broker. The broker initiates nothing.
    { "action": "accept",
      "src": ["tag:watcher", "tag:archiver", "tag:replicator"],
      "dst": ["tag:broker:6379"] },
    // Existing, unchanged — narrows in meaning as the other two leave this node.
    { "action": "accept", "src": ["tag:watcher"], "dst": ["tag:notifier:9000,9001"] },
    { "action": "accept", "src": ["autogroup:member"], "dst": ["*:*"] }
  ]
}

Plan

Phase 1 — Bus client hardening (independent; do first)

No infrastructure dependency. Everything here is worth doing even if D2 never happens.

  1. Give src/core/bus.py:147 an explicit connection policy: socket_connect_timeout, socket_timeout, health_check_interval, retry_on_timeout. TDD, per methodology.
  2. Establish what AsyncBusTailReader does when the connection drops mid-tail — resume from last id, restart at 0-0, or stop silently. Restarting at 0-0 is correct but expensive (apply-iff-generation > makes replay idempotent, Consume registry announcements and reconcile watched_items; drop the Archiver SDK #254); stopping silently is the failure that looks like an empty registry, which the same doc calls out as the mistake that fails silently. Test the answer, don't assume it.
  3. Same question for the content.blobs group consumer, where XREADGROUP should resume cleanly — verify rather than assume.
  4. Make a broker partition loud. WATCHER_BUS_REDIS_URL unset already skips loudly; a set URL that is unreachable must be equally loud and must not present as an idle system.

Phase 2 — Provision the broker node (blocked on prerequisites)

  1. new --name broker via POST https://exe.dev/exec. Region lax to sit beside watcher, its highest-frequency participant — and note the region caveat above.
  2. Setup script: sudo-prefixed throughout, logging to a user-writable path. notifier#43's P1 — the script began exec >>/var/log/…, which the unprivileged exedev user cannot open, and under set -euo pipefail that aborted before anything ran, leaving the auth key unshredded in /exe.dev/setup.
  3. exeuntu ships Tailscale pre-installed; only tailscaled needs enabling. The curl | sh in notifier's script was never necessary.
  4. Join as broker with the single-tag key via --auth-key=file:, then shred.
  5. Install and configure redis-server: tailnet bind, requirepass/ACL users per D7, and a persistence policy the broker owner is happy with (appendonly no today).
  6. The broker's own boot race is real and is R1 from notifier#43 verbatim — a tailnet-bound Redis started before tailscaled assigns the address is precisely observo#473, which cost two weeks of silently-starved jobs. After=tailscaled.service plus a wait that probes /proc/net/fib_trie, never ip addr (observo#479: the systemd sandbox SIGSYS-kills ip and blocks AF_NETLINK silently, so the probe exits 0 and detects nothing). Redis 7's - bind prefix makes an optional bind possible here — unlike uvicorn — but an optional bind that silently comes up loopback-only is the wrong trade for a broker. Fail loudly.

Phase 3 — Cut watcher over to the remote broker

  1. Point WATCHER_BUS_REDIS_URL at redis://:<secret>@broker:6379/0. Keep WATCHER_BUS_ENABLED=1 in the unit — both halves assigned, or startup aborts (Gate the bus producers behind a unit-only flag, as ARCHIVER_BUS_CONSUMER does for the consumer #262).
  2. Drop After=…redis-server.service / Wants=redis-server.service from deploy/watcher.service. The soft-ordering rationale in that comment block is about a local broker and stops being true; replace it rather than deleting it silently.
  3. Extend tests/deploy/test_installed_unit_matches_repo.py to pin the new invariants.
  4. Update scripts/dev_server.sh and tests/conftest.py clearing behaviour for a remote WATCHER_DEV_BUS_REDIS_URL — a scratch bus is now a scratch database on a remote broker, and the Gate the bus producers behind a unit-only flag, as ARCHIVER_BUS_CONSUMER does for the consumer #262 hazard is unchanged in kind but wider in blast radius.
  5. Measure round-trip publish latency and the content.blobs consumer's steady-state behaviour over the tailnet. notifier's lax→pdx was 31 ms direct P2P after a DERP-relayed cold start at 85 ms — the cold path is the number that matters for a consumer loop waking after idle.

Phase 4 — Archiver and replicator leave (external dependency)

  1. Tracked in their own repos, not here. Watcher's side is: confirm every stream it publishes still has its consumer, and that the ACL admits both directions.
  2. As each leaves, remove its database and role from this cluster: archiver, archiver_test, role archiver — replicator has none. Remove the departed venvs (regenerate with uv sync), and /var/lib/replicator/blobs. Retain both checkouts per D8.
  3. Verify tag:watcher now describes one service.

Phase 5 — Retention (D5; independent, do early)

  1. Decide a retention window for procrastinate_events and procrastinate_jobs. Procrastinate 3.7.2 ships JobManager.delete_old_jobs (procrastinate/manager.py:247); nothing in this repo calls it.
  2. Add it to the weekly scripts/cleanup.sh path so the 301 MB cannot silently rebuild.
  3. Reclaim: a one-off VACUUM FULL after the first prune. Check free space first — VACUUM FULL rewrites the table and needs room for the copy, and there is 1.1 G free. Delete rows first, vacuum second; pg_repack if the margin is uncomfortable.
  4. Record the actual reclaim honestly against the 95 % starting point, per notifier#43's closing note.

Phase 6 — Docs

  1. AGENTS.md: Infrastructure (the service/port table describes a VM that will no longer exist), Server Lifecycle, The bus (broker is no longer "on this VM"), Environment Files.
  2. docs/ARCHITECTURE.md → Redis and the bus: "Archiver operates redis-server on this VM" becomes a neutral node; Sibling services: archiver is no longer co-located.
  3. docs/ENVIRONMENT.md: WATCHER_BUS_REDIS_URL row still reads redis://localhost:6379/0.
  4. docs/COMMANDS.md: the Archiver-sibling test setup survives D8 — say so explicitly, because "archiver moved off this VM" reads like the checkout went with it.
  5. docs/DEPLOYMENT.md, README.md, and src/core/db_safety.py's docstring. Note what notifier#43 got right here: the dev-and-prod-on-one-box arrangement that makes alembic's carve-out necessary does not change — what stopped being shared is the host, not that.
  6. New docs/reference/tailscale.md describing this node, deferring the general primer to observo's copy.

Risks

R1 — the bus client has no failure policy, and the single process makes that structural. src/core/bus.py:147 is a bare Redis.from_url with library defaults. Loopback masked it completely. On a WireGuard hop, a stalled read has no bound, and because one uvicorn process runs the API and the content.blobs consumer and the embedded worker, a wedged consumer coroutine can sit inside a process whose /health stays green. This is the highest risk in the epic and the reason Phase 1 runs first and independently. Notifier's equivalent — NotifierClient's 10 s timeout — already existed before its migration; watcher's does not exist yet.

R2 — info.registry is groupless and replays from 0-0 every boot. The design is deliberate (#254): reading from $ fails silently, showing a booting worker an empty registry indistinguishable from a genuinely empty one. A remote broker adds reconnects, which is a new way to re-enter that path mid-life rather than at boot. Replay is idempotent under apply-iff-generation >, so the correctness risk is low and the cost risk is real — full replay on every blip. Phase 1 step 2 is the gate.

R3 — content.fetch crossing a network means fetches can stop quietly. A partition stops command publication; replicator falls back to its conservative default pacing and nothing obviously breaks. The symptom of a broken bus is a slow system, not a failing one. Phase 1 step 4 exists for this.

R4 — this epic's completion is not in this repo. Phases 4 and 6 cannot close until archiver and replicator have migrated, each of which is its own epic in its own repo under the cross-repo policy. Watcher's Phases 1, 2, 3 and 5 are independently deliverable; the tracking issue should not read as blocked on them.

R5 — VACUUM FULL on a 95 %-full disk. 183 MB + 118 MB of table to rewrite with 1.1 G free. Order matters: delete first, vacuum second, check df between.

R6 — D3 and D4 together mean the exe.dev public/private bit is the entire boundary on an unauthenticated admin UI. One share set-public — deliberate or mistaken — puts every dashboard route, mutations included, on the open internet, and nothing local would notice. Verify the current setting (ssh exe.dev share …) as part of this epic rather than assuming it, and record the result. Accepted per D3/D4; recorded so it is a known position rather than a discovery.

R7 — the disk expectation, measured rather than assumed. notifier#43's closing note applies directly, and the numbers here say the same thing:

What leaves Size
replicator/.venv 181 MB
archiver/.venv 131 MB
DBs archiver + archiver_test 18 MB
/var/lib/replicator/blobs 2 MB (backend is gcs; local dir is spillover)
Total ~330 MB
Phase 5 retention, for comparison ~300 MB

Replicator has no database in this cluster at all — it holds Redis plus a GCS blob backend. So the departures and the prune are worth roughly the same, and both together take a 19 G disk from ~1.1 G free to ~1.7 G — 95 % → ~91 %. Real, and not a fix. Do not plan this epic around the split solving the 95 %.

R8 — no rollback rehearsal is possible for the bus cutover. notifier's rollback was two URLs and a restart, with the old data untouched. Reverting Phase 3 means repointing WATCHER_BUS_REDIS_URL back at a local broker that, by Phase 4, no longer exists. Sequence Phase 3 before Phase 4's decommission, and keep the local broker running until the remote one has carried real traffic.

Success criteria

  • src/core/bus.py builds its client with explicit timeouts, health checks and a retry policy, test-guarded
  • Reconnect behaviour is established by test for both consumers — content.blobs (grouped) and info.registry (groupless, 0-0)
  • An unreachable-but-configured broker is loud, and does not present as an idle system
  • Broker node is tagged, non-expiring, tailnet-bound, password- or ACL-protected, and survives a reboot with NRestarts=0
  • Watcher publishes all four streams and consumes both across the tailnet, with cold-path and warm-path latency measured and recorded
  • tag:watcher describes exactly one service
  • procrastinate_events + procrastinate_jobs under a standing retention policy invoked by scripts/cleanup.sh; reclaim recorded honestly against 95 %
  • ARCHIVER_REPO_PATH still resolves and the full suite is green, archiver's service having left
  • The exe.dev proxy's public/private setting for this VM verified and recorded
  • Docs describe a watcher-only VM and a neutral broker

Follow-ups (separate issues)

  • Archiver repo — its own migration epic, plus the broker-ownership handover implied by D2. Cross-repo policy: recommend, get approval, file there.
  • Replicator repo — its own migration epic.
  • Dashboard authentication — deliberately out of scope per D4. Worth its own issue on the merits, independent of any VM boundary.
  • Broker persistence and monitoring policy — appendonly no today; the neutral node's owner should restate it deliberately rather than inherit it.

Pattern and hard-won operational lessons inherited from notifier#43. Watcher's side of that migration closed in #280.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions