fix(backup): stop shipping the archives back through the log pipeline (#286)

[Gerrrt]

Closes #286.

backup-volumes.sh archives each volume with docker run ... tar -czf -, so
the gzip stream is that container's stdout. The json-file driver records
stdout whether or not anyone reads it, and Alloy's loki.source.docker tails
every container on the socket — so make backup sent its own archives back
into Loki as log lines.

ingested in ~3 min, 2026-08-29	765 MB
peak rate	2.7 MiB/s vs 950 B/s baseline
peak line rate	5,500/s vs 3/s
loki RSS	100 → 1015 MiB → 107
per run, vs the estate's real logs	~3 days' worth

It also quietly undid archive_one()'s own argument: that function encrypts
with age precisely so the plaintext never touches this disk, and the
compressed plaintext was landing in loki-data, which is not encrypted, for
the 30-day retention.

The fix #286 proposed would have broken log collection

The issue said to relabel on
__meta_docker_container_label_com_docker_compose_project and drop what does
not carry it. I checked that against container_last_seen on both hosts before
writing any of it:

oracle       wiki                 project=<NONE>
oracle       db                   project=<NONE>
oracle       alloy                project=<NONE>
prometheus   grafana              project=observability
prometheus   loki                 project=observability   ...

Everything on oracle is a plain docker run (scripts/deploy-agent.sh), so
that filter would have silently stopped collecting logs for the one service
in this estate anybody uses — the #62 shape exactly: no error, no failed
component, just an empty dashboard. An allow-list of known-good names is the
same trap one step further on; it fails closed on a host nobody remembered.

What landed instead

1. --log-driver none on the runs whose stdout is a data stream. Docker
   discards it at source — nothing to tail, nothing to filter. Nothing reads
   docker logs on a container the shell is already piping.
2. An opt-out label, homelab.logs=off, dropped in discovery.relabel
   before Alloy opens a stream. Second line of defence for the next ad-hoc
   docker run that forgets.

Applied to the archiver runs in backup-volumes.sh and restore-volumes.sh,
including the pre-restore safety snapshot, which is the same tar-to-stdout
shape and would have done the same thing.

What this does not fix

Opt-out is weaker than the allow-list #286 wanted, and the gap is in the config
comment rather than papered over: a future throwaway container setting neither
flag is still tailed. What bounds that is a per-stream ingestion limit in
Loki (limits_config sets none today, so the 2.7 MiB/s flood sailed under the
4 MB/s default). That trades a flood for silently dropped lines, which is its
own decision and not this PR's.

loki's mem_limit is deliberately unchanged

#114 sized it at 1536M from a 1015 MiB peak that was this bug. Every
measurement behind that number still includes the flood, and a fortnight
without one does not exist until 2026-09-18. The comment in compose.yaml
now carries that date and what to re-derive from — the nine quiet days in the
window peaked at 107-128 MiB, so expect roughly 512M.

Moving it now would be sizing from a window I have already established is
contaminated, which is the mistake #114 spent two weeks not making.

Verification

make validate passes. Post-merge I will deploy and check, positively in both
directions, that an unlabelled container is still collected and a labelled one
is not — then run a real make backup and measure. Results in a comment.

🤖 Generated with Claude Code

[Gerrrt]

Deployed and verified

make converge → 80202ae1b6bf, behind=0 dirty=0 verified=1 apply=1.

Both directions, because "no flood" and "collection is broken" look identical

That is the whole risk in this change, and it is the #62 shape: a filter that
drops too much produces silence, and silence is what success looks like. So it
was checked positively as well as negatively, with two probe containers
differing only in the label.

Read out of discovery.relabel.dockerlogs_targets — its input against its
exports, which is the distinction that matters:

arguments (INPUT):   logtest-drop, logtest-keep, logtest2-drop, logtest2-keep
exports  (OUTPUT):   logtest-keep,               logtest2-keep

And in Loki, with both probes emitting every 20s:

stream	lines in Loki
logtest2-keep-… (no label)	3, climbing
logtest2-drop-… (homelab.logs=off)	0

The unlabelled container is still collected. Only the labelled one is dropped.

A real make backup

backups/volumes/20260904T232922Z — 5 volumes, 1.1 GB written, stack down
107s. Everything that reached Loki across the whole 149-second window:

/var/log/syslog        191.8 KiB      prometheus           8.7 KiB
loki                   121.2 KiB      alertmanager         1.3 KiB
/var/log/journal       121.0 KiB      /var/log/auth.log    0.4 KiB
alloy                   44.2 KiB      db                   0.3 KiB
grafana                 42.8 KiB      blackbox-exporter    0.3 KiB
/var/log/*.log          42.7 KiB      suricata             0.2 KiB
filterlog               11.5 KiB      syslogd              0.0 KiB
                                      TOTAL              586.7 KiB

No archiver stream at all — no random-named container, nothing binary. Every
line there is ordinary syslog, journal, and the services' own logs. Against
765 MB on 2026-08-29.

	before	after
archiver bytes into Loki per run	765 MB	0
peak ingest rate	2769 KiB/s	77 KiB/s
loki peak RSS	1015 MiB	229 MiB

0 OOM kills; all seven services back up with their limits intact.

One number reported straight

Loki's RSS is sitting at 226-229 MiB, not the 107-128 MiB of the quiet days in
the #114 window. That is 4.4x below the flood and 15% of its 1536M ceiling, so
nothing here is at risk — but it is not "within the routine range" that #286's
done-when asked for, and it would be easy to write it up as though it were. The
likely cause is the post-quiesce restart rather than the backup: the stack was
stopped for 107s and Loki replays its WAL and reopens its index on the way back.

It is worth watching rather than explaining away, and the 2026-09-18
re-derivation of mem_limit will settle it either way — if the steady state is
really ~226 MiB rather than ~120, the right ceiling is nearer 768M than 512M and
the comment in compose.yaml should say so.

Still open, by design

• loki's mem_limit stays 1536M until 2026-09-18. Sizing it today would
  mean sizing from a window that contains the flood.
• No Loki per-stream ingestion limit. That is what would bound a throwaway
  container which sets neither flag, and it trades a flood for silently dropped
  lines — its own decision, named in docker.alloy and not taken here.