watcher: event-driven discovery (FSEvents) — polling measured out at ~2.4s/poll, gate needs <5%

[EtanHey]

Why this exists

R3 (#759) fixed the two defects responsible for the JSONL watcher's idle CPU burn and took
poll_once from 18.0s → 2.4s (7.4×). It still does not clear the <5% idle-CPU gate,
and com.brainlayer.watch remains => disabled.

This issue records exactly how far polling can go, so whoever picks it up starts from evidence
rather than re-deriving it. Everything below was measured on the M4 against the real corpus
(~12,800 JSONL files, 21,529 registry entries), nice -n 15 taskpolicy -b.

The polling floor, measured

stage	poll_once, steady state	duty @30s
before R3	18.0s	60%
after prune-retry fix	5.0s	17%
after denylist memo (shipped in #759)	2.4s	8.3%

Of that remaining 2.4s, discovery is 2.19–2.56s — a **/*.jsonl walk across 5 provider roots
plus a stat on every file. A standalone bare glob+stat sweep of the same corpus measures
~1.03s, so ~1s is the hard floor of any polling design here. There is no further meaningful
win left inside the poll loop.

Soak results (10 min, sampled every 10s with ps, externally — never the watcher's own health file)

run	code	mean idle CPU	verdict
1	pre-change-detector	4.88%	pass by 0.12
2	shipping (#759)	6.41%	FAIL

Near-identical code, same method, same machine — 1.5 points apart. max ≈ 98.6% in both,
because the cost is a ~2.4s burst per poll, not a smooth trickle.

The load-bearing conclusion: run-to-run variance is as large as the margin. A single 10-minute
soak cannot resolve a 5% gate on a design that sits at 4–6.5%. Re-running until one passes is
selection, not evidence.

Why a longer poll interval is the wrong fix

Raising the interval divides the floor by a bigger number. It buys margin with ingestion latency
and still leaves the burst — and at the variance above it does not buy reliable margin.
--poll 60 was tried during R3 and the shipping code still measured 6.41%.

What to build

Event-driven discovery via FSEvents — the first branch of the original R3 constraint
("event-driven, or batched at ≥30 s"), which the polling branch has now been measured to the end
of. The watcher should be woken by the filesystem instead of asking it ~2,880 times a day whether
anything changed.

Design notes from what R3 touched:

• Discovery and read are already separable. _discover_jsonl_files() finds new files; the
  (mtime, size) skip gate added in fix(watcher): stop the idle burn (prune re-scan + 4x denylist), and give the watcher a heartbeat #759 already makes already-tracked, unchanged files free. An
  FSEvents stream supplies exactly the "what changed" signal the sweep is currently brute-forcing.
• Keep the skip gate's refusal conditions. They exist because a false skip stalls a session
  silently: no tailer yet, tailer.offset < size (a read capped by max_lines_per_file leaves the
  stat unchanged but the tailer short), a complete line still buffered, pending
  quarantine/ingestion failures. Any event-driven path needs the same guards on its fallback sweep.
• Keep a slow full sweep as a backstop. FSEvents can drop events (volume unmount, coalescing,
  historical replay gaps). A periodic reconciliation sweep — minutes, not seconds — is the safety
  net, and at that cadence its ~2.4s cost is negligible.
• The prune interacts. prune_missing_files is now retried on a parent-dir change-detector plus
  a 900s timer (BRAINLAYER_WATCHER_OFFSET_PRUNE_RETRY_S). It is deliberately unable to prune
  entries whose root is unmounted; do not "fix" that guard.

Acceptance

• 10-minute idle soak repeated at least 3 times, mean idle CPU <5% in every run — one run
  is not a measurement at this variance.
• Sampled externally (ps), never from the watcher's own health file: a soak the watcher grades
  itself on inherits whatever probe defects it has.
• Measured from a caught-up offset registry, so it is idle cost and not backlog catch-up.
  Reporting catch-up CPU as idle CPU is the specific false green R3 hit twice.

Not in scope here

com.brainlayer.watch re-enable is a separate decision currently with Etan: accept ~5–6.5% and
enable now, leave it down until this lands, or something else. This issue is the engineering
path, not that decision.

Related: #759.

— brainlayerClaude-e358a78e (worker) · claude-code/claude-opus-5