brainbar_hybrid_helper wedges to 100% of a core every ~15-20 min; the wedging query is still unidentified

[EtanHey]

Problem

brainbar_hybrid_helper wedges to 100% of one core and stays there until killed. Observed three
times on 2026-08-24 alone: 19:04, 19:20, 20:14 — roughly every 15–20 minutes, unprompted. The
first instance had burned 46m12s of CPU across ~7h of life.

Evidence it is a wedge, not load

SIGTERM the wedged process and BrainBar respawns it against the same DB and the same client
traffic — the replacement measures 0% CPU. Same code, same load, 100% → 0%. A loaded process
would not do that.

sample <pid> 5 puts 4150/4150 busy-thread samples in sqlite3_step → sqlite3VdbeExec, dominated
by sqlite3VdbeMemSetText / sqlite3VdbeMemGrow (row-by-row text materialization) plus
sqlite3BtreeIndexMoveto, with no fts5 symbols anywhere in the hot path. That is a scan over
chunks (1.07M rows / 14.8 GB), not an FTS hit or an index seek.

What PR #735 does and does not do

#735 adds a hard request deadline so a wedged query can no longer peg a core indefinitely — it logs
the offending request and recycles the process. That is containment, not a cure. This issue tracks
the remaining work: identify the actual query.

Two real leads (neither proven to be this wedge — do not assume)

1. src/brainlayer/mcp/search_repo.py:2117 —
   fts_timeout_ms = _BRAINBAR_HELPER_FTS_BUDGET_MS if brainbar_helper_fast_profile else None.
   FTS is bounded only for the BrainBar fast profile; every other caller runs FTS unbounded.
2. search_repo.py:2266 — the trigram fetch is the one _fetch_fts_rows call that passes no
   timeout_ms at all, while its siblings on :2263/:2265 do.

⚠️ Constraint for whoever takes this

_fetch_fts_rows returns [] on timeout. So "just add timeouts everywhere" would silently drop
search results, which BrainLayer forbids. Fix the unboundedness without introducing silent
truncation — #735 deliberately fails loudly for this reason.

Acceptance criteria

• The wedging statement is identified with evidence (not inferred).
• It is bounded or made index-satisfiable without silently truncating results.
• The helper survives 24h under normal fleet traffic with no 100%-core episode.

— brainlayerClaude (lead) · claude-code/claude-opus-5

[EtanHey]

Cadence data, including the sample that contradicts my own headline

PR #735 merged (a52aa855) adds the request deadline. It is MERGED-NOT-DEPLOYED: the helper runs from the brew cask venv (/opt/homebrew/Cellar/brainlayer/1.5.9/libexec/venv/...), and scripts/launchd/install.sh installs launchd plists and helper scripts — never the Python package. The fix goes live only with a v1.5.10 release + brew upgrade, which is the first act of the next sprint.

Observed wedges, 2026-08-24

#	time	evidence
1	19:04	pid 1216 — 100% CPU, 46m12s of CPU burned over ~7h of life
2	19:20	pid 89116 — 0% at 3m43s of life, then 99% sustained (cputime delta 244.9s → 256.8s over 12s wall)
3	20:14	pid 78681 — 98% sustained; 7m26s of CPU in 19m57s of life

Each was relieved by SIGTERM; BrainBar respawned a warm replacement in ~9s each time.

⚠️ The counter-sample — my "every 15–20 minutes" is probably wrong

pid 53329 ran 36m31s with its cputime frozen at 306s and 0% CPU at the end of the window. That is well past the interval the three samples above suggested.

What differs: samples 1–3 were taken while the machine was heavily loaded (load average 21.6 → 8.4; hotlane at 120%, the Claude statusline spawning paired full-table COUNT(*) scans of a 14.8 GB DB on every keystroke render across 7–8 panes). The 36-minute clean run happened after all of that was stopped, with load at 1.61.

So the honest reading is the wedge may be load-triggered rather than periodic — plausibly a timing/contention effect (SQLite busy contention, or a query that only becomes pathological when starved) rather than a clock. I am recording this explicitly because my earlier "recurs every 15–20 min" framing was drawn from three samples in one hot window, and someone planning work off it deserves the caveat rather than the headline.

Implication for whoever picks this up: do not expect to reproduce it on an idle box. Reproduce it under load, and capture the statement before killing the process — that is the missing evidence this issue is actually about.

— brainlayerClaude (lead) · claude-code/claude-opus-5