fix: content the pipeline was silently losing (articles, ASR, Firecrawl) + verify coverage on the dashboard - #133
Open
VGonPa wants to merge 7 commits into
Open
fix: content the pipeline was silently losing (articles, ASR, Firecrawl) + verify coverage on the dashboard#133VGonPa wants to merge 7 commits into
VGonPa wants to merge 7 commits into
Conversation
…t, video X serialises a long-form Article as ordered `blocks` + `entityMap`. The parser only understood two shapes: plain `unstyled` text and an `atomic` block whose entity is an IMAGE. Everything else was silently discarded, so an Article that used any richer construct reached the vault with holes in it — and nothing anywhere said so. Four kinds of content were being lost: * MARKDOWN / TEXT / HTML entities — the paragraph body lived under a different key than the one being read, so the block rendered as empty and vanished. * DIVIDER — a section break with no text; dropped, silently merging two sections into one. * TWEET — an embedded post; dropped entirely, taking with it the only pointer to what the paragraph was talking about. Now rendered as its status URL. * MEDIA/VIDEO — an embedded video; the atomic block was recognised, failed the IMAGE check, and was dropped. Now a first-class `ArticleVideoBlock`. Built against 29 real captured payloads rather than a guess at the schema, which is the only reason the `bit_rate` vs `bitrate` bug surfaced: X spells the key one way, `select_variant` reads the other, so every video silently resolved to the LOWEST-quality mp4. `_normalise_variant` renames it at the boundary. A video entity with an empty `variants` list now yields no video block at all. Trusting it produced a "video" whose URL was the poster JPEG — a broken embed that looked like a working one. `_warn_if_media_unresolved` reports drift between the entities in a payload and the blocks that came out, so the NEXT schema change is loud instead of silent. It requires a MEDIA/IMAGE entity before it fires; counting any `atomic` block made every prose+code article warn about media it never had. `generate` renders a pending article video as a visible link line rather than nothing. A video that has not been downloaded yet is a fact about the pipeline; staying silent about it is how content goes missing without anyone noticing. The dispatch now closes with `assert_never`, so a future block kind fails to type-check instead of falling into a catch-all and disappearing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rakeet parakeet-mlx is English-only, and on Spanish audio it does not fail — it FABRICATES fluent English. That is the worst failure mode available: a confident transcript nobody can tell is invented, feeding the video digest and every summary downstream. This corpus is bilingual, so the wrapper had to stop being a single backend. `xbrain-transcribe-auto` slices the first 30s with ffmpeg, runs whisper with NO `--language` to detect, and dispatches on the answer. English → parakeet. Anything else → whisper. Detection failed, returned nothing, or is ambiguous → whisper. The asymmetry is deliberate: whisper on English audio costs time, parakeet on Spanish audio costs truth. Backend order for non-English is `xbrain-transcribe-mlx` first, falling back to `xbrain-transcribe-whisper`. Measured on the same 68s clip: brew whisper 7m25s on CPU vs mlx-whisper 17.7s on the Apple GPU — 25x — with byte-identical text. The mlx wrapper exits non-zero WITHOUT writing an output file when the model returns nothing, so the router's fallback actually triggers instead of the caller reading a half-written file as success. Detection model is `base`, not `tiny`. Measured: ~19s vs ~10s, both correct on the clips tested. Nine seconds is not worth a coin-flip on which backend runs. `xbrain-transcribe-whisper` joins the repo (it was loose on the machine) and gains silence-hallucination handling: whisper emits subtitle-credit boilerplate over near-silent audio, and ffprobe cannot catch it because the file DOES have an audio stream. One such fabrication was already in the vault, stored with has_speech=true. The match is on the whole stripped transcript only — a substring match would delete real speech that happens to contain the phrase. `config.toml` is gitignored, so the wiring is documented in `config.toml.example` where someone can actually find it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The key was only ever read from `$FIRECRAWL_API_KEY`. Anyone who had logged in
with the Firecrawl CLI — which writes a credentials file — had a working key on
disk and still got the "no key configured" path, so `fetch --retry-failed`
quietly fell back to the weaker extractor. The failure was invisible: fewer
articles recovered, no error, nothing to grep for.
`firecrawl_key()` now falls back to the CLI credentials file, checking the
macOS, XDG and legacy locations in order. A malformed or unreadable file
resolves to None rather than raising: the credential is an optional
enhancement, and a broken one must degrade exactly like an absent one.
`XBRAIN_NO_FIRECRAWL` still wins over both, so the escape hatch keeps working.
`_echo_retry_plan` now prints WHERE it looked when it finds nothing. "No key"
is not actionable; "no key in $FIRECRAWL_API_KEY, none at <these three paths>"
is.
The autouse fixture in conftest is the load-bearing part of this change. The
existing "no key configured" tests asserted the fallback path — and once the
code reads a real file, those tests would read the DEVELOPER'S machine and pass
or fail depending on whether that person happens to have logged into Firecrawl.
The fixture points the lookup at a path inside tmp_path, so the suite tests the
code instead of the laptop.
The literal in the new tests is a fake ("fc-padded"), audited into the
detect-secrets baseline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dashboard had no verification signal at all. After a verify pass the only place the result existed was a markdown report on disk, so the dashboard kept showing "Enriched 100%" with nothing anywhere distinguishing enriched from CHECKED. Those are very different claims. Panel 09 draws ONE full-width stacked bar across the whole population — PASS / REVIEW / FAIL / UNJUDGED — so the judged slice can only ever be read against the total. A verdict mix on its own would report "96% PASS" off 51 judgments out of 4,843 outputs, which is true and deeply misleading. The caveat line under the chart states the unjudged count in words, because a legend entry is easy to skip and a sentence is not. Coverage counts OUTPUTS, not items: one post carries a summary, a topics assignment and possibly a video digest, each judged separately against its own source. A stale verdict counts as unjudged, not as its stored verdict. Re-generating a judged output invalidates it — the verdict describes text nobody reads any more. This is not hypothetical: the two FAILs from the last pass went stale the moment their summaries were fixed, and showing them as current failures would be as wrong as showing them as passes. Stale is surfaced separately, with its own count, because "70 verdicts need re-earning" is actionable and folding them into unjudged would hide the debt. `compute_dashboard_data` now takes the output language: a verdict is bound to the rubric contract it was judged under, and that contract is per-language. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`test_ruff_examines_every_python_file_in_the_repo` caught all three new scripts: ruff does not discover extensionless `#!/usr/bin/env python3` executables from a directory, so `ruff check scripts` was walking straight past them. The gate would have printed ALL CRITICAL CHECKS PASSED over three unlinted files. Listing them in `extend-include` turned up 3 real E702s that had been invisible, now split, plus formatting the formatter had never been allowed to see. This is the fail-closed gate doing exactly what it was built for, on the first new files added after it landed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…body could read
`scan_store` was fixed once to KEEP records whose only candidates are uncertain.
`summarise_scan` then dropped them again, and `render_entity_report` filtered
them out of the table, so neither the terminal line nor the report showed a
single one. The docstring on `EntityRecord` promised "never dropped"; the code
dropped them two layers later.
This is not academic. A candidate lands in the uncertain tier when its POSITION
makes the capital ambiguous — chiefly sentence-initial, where Spanish
capitalises the first word whether or not it is a name. That is exactly where
this corpus puts names: summaries open with the subject ("Suhail afirma…",
"Ethan Mollick destaca…"). The one fabricated name the LLM judge pass caught —
"Suhail", where the source says @shl (Sahil Lavingia) — sits in this tier,
ungrounded, and the check had already found it:
extract_entities("Suhail (shl) afirma…") -> ['PRs']
extract_uncertain("Suhail (shl) afirma…") -> ['Suhail'] is_grounded -> False
The finding existed. No reader could reach it.
Reported on its own line and in its own table section, never merged into the
headline: the tiers have very different precision, and one number covering two
instruments is how a reader ends up quoting the wrong one. The gate still counts
confident flags only — that split is deliberate and unchanged.
On the corpus this makes visible 1,461 summaries and 117 digests that were
silently set aside.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ollision Precision of the confident tier was measured at 32% on a 50-flag sample (the report had said it "has NOT been independently re-measured"). Ruling on each flag turned up two real defects and one the fix itself introduced. **ASR disfluencies inside a name.** The evidence for `Application Default Credentials` reads "the application default uh credential". All three words are there and adjacent; a name is matched as a contiguous window, so one filler means no window of the transcript ever spells it. Stripped from the EVIDENCE side only — the output is written text and has no fillers, so stripping there would corrupt real names. **Translated names arrive reversed.** Spanish puts the adjective after the noun: `Revolución Industrial` against "the industrial revolution". Reversal alone is not enough, because one word is still in the other language — it composes with the existing fuzzy threshold, which absorbs "industrialrevolucion" vs "industrialrevolution". No lexicon entry, and it generalises to pairs the corpus has not produced yet. **A short form collides with the joint of two ordinary words.** Found by a test written expecting it to pass: adding `usa` to the lexicon made the window `us`+`a` of "gives us a much better result" ground `Estados Unidos` off a pronoun and an article. Squashing erases word boundaries, so forms of ≤3 chars now match a single token only — the same reasoning as `_FUZZY_MIN_LEN`. `us` stays OUT of the lexicon. Grounding the country off the English pronoun, present in nearly every transcript, does not remove a false alarm — it blinds the check to every invented mention of the US. 8 flagged sources whose only evidence is bare `us` stay flagged, on purpose, with a test pinning it. Effect, measured rather than projected: 175 → 160 flagged entities on summary, UNCHANGED on digest, and sample precision 32% → 34.8%. I had projected 55-60%. Two of the three false-positive classes turned out smaller than they looked: `España` and `Europa` were not translation misses at all (the evidence genuinely lacks the name, or carries only the adjectival form), and `ChatGPT Plus` was the check working correctly on non-adjacent words. The dominant remaining class is untouched: the extractor pulls Spanish common words as entities (Zona, Entrenar, Cuello, CEO, Pro). That is also what makes the uncertain tier unusable, so one filter fixes both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five commits, one theme: the pipeline was dropping content and saying nothing about it.
What was being lost
Article blocks. X serialises long-form Articles as Draft.js
blocks+entityMap. The parser understood exactly two shapes — plain text and anatomicIMAGE — and silently discarded everything else: MARKDOWN/TEXT/HTML paragraph bodies, DIVIDER section breaks, embedded TWEETs, and embedded VIDEOs. Built against 29 real captured payloads, not a guess at the schema, which is the only reason thebit_ratevsbitratebug turned up: X spells the key one way andselect_variantreads the other, so every article video was silently resolving to the lowest-quality mp4.Transcripts.
parakeet-mlxis English-only and does not fail on Spanish — it fabricates fluent English. On a bilingual corpus that is the worst failure mode available: a confident transcript nobody can tell is invented, feeding the digest and every summary downstream.xbrain-transcribe-autodetects the language on the first 30s and dispatches; any uncertainty goes to whisper, never to parakeet. Also fixes a real hallucination already in the vault (whisper's subtitle-credit boilerplate over near-silent audio, stored ashas_speech=true, which ffprobe cannot catch because the file does have an audio stream).Articles that could have been recovered. The Firecrawl key was read only from
$FIRECRAWL_API_KEY, so anyone logged in via the Firecrawl CLI had a working key on disk and still got the fallback extractor. No error, fewer articles, nothing to grep for.And one thing that was being over-claimed
The dashboard showed
Enriched 100%with no way to tell enriched from checked. Panel 09 now draws one full-width bar across the whole population — PASS / REVIEW / FAIL / UNJUDGED — so the judged slice can only be read against the total. On the current corpus that is 51 of 4,843 outputs: 1.1% coverage, which is the honest number and looks nothing like "96% PASS".Stale verdicts count as unjudged, not as their stored verdict. Re-generating a judged output invalidates it, and this is not hypothetical — the two FAILs from the last pass went stale the moment their summaries were fixed.
Notes for review
tests/conftest.pygains an autouse fixture that points Firecrawl credential lookup attmp_path. Without it, the existing "no key" tests would read the reviewer's own machine and pass or fail depending on whether they happen to be logged into Firecrawl.#!/usr/bin/env python3files from a directory, so it had been walking past them. Listing them surfaced 3 real E702s.config.tomlis gitignored, so the ASR wiring is documented inconfig.toml.example.Gate: 1724 passed · ruff clean · mypy clean on 48 source files.
🤖 Generated with Claude Code