Index Discord links alongside the newsletter, deduped by aashutoshrathi · Pull Request #8 · The-Nibble/dig

aashutoshrathi · 2026-08-29T05:29:57Z

Indexes the links shared across the Discord server alongside the newsletter, deduped into one archive.

1,635 Discord-only links join 1,836 newsletter entries. 112 links turned out to be in both — and 98 of those appeared in the channel before the edition that ran them. That last number is the point of merging rather than concatenating: the archive can now show that the community found something months before it was written up.

How it fits together

build-index.py needs a Substack export that only exists on a laptop, so CI could never run it. The build is split:

Substack export --build-index.py-->  data/nibble.json  ----.
                                                           |
Discord --fetch-discord.py-->  data/discord.json  ---.     |
                                     |               |     |
                 enrich-links.py -->  data/link-meta.json  |
                                                     |     |
                              build-page.py  <-------'-----'
                                     |
                              index.html

build-page.py is the only writer of index.html and is pure — same inputs, byte-identical page — which is what lets the daily job rebuild without the archive. The harvest is stored losslessly and every heuristic runs at build time, so a bad rule is never baked into the data and widening a filter never needs a re-fetch.

Channels

Set DISCORD_GUILD_ID and discovery finds the channels itself — every text and announcement channel @everyone can view, plus threads and forum posts. 38 channels and 71 threads on this server. A channel created next month is indexed the next morning with no config change. Private channels and private categories are filtered by the @everyone View Channel bit; anything the token can't read is skipped with a note rather than failing the run.

Each channel and thread keeps its own cursor, and the cursor advances past every message seen rather than every message kept — otherwise a run of link-free chat would be re-fetched forever. Archived threads are enumerated once per parent, since an archived thread can't gain a message without being unarchived, which puts it back in the active list.

HIDE_CHANNELS keeps #memes, #liked-phrases, #introductions and #job-posts off the page without dropping them from the harvest.

Privacy

data/discord.json is committed to a public repo, so it stores only the commentary around a link, never the message it arrived in. Message text drops from 146 KB to 40 KB, and the 1,316 messages that were a bare link store nothing at all. Account ids aren't collected — nothing downstream read them.

Naming the links

A newsletter entry arrives already named. A Discord message is usually a bare url, so enrich-links.py fetches OpenGraph tags, with dedicated APIs where a scrape does worse: GitHub, YouTube oEmbed, Wikipedia REST, arXiv Atom, the HN Firebase API (a link to a comment climbs to its parent story), and fxtwitter for posts. Cached by canonical url and committed, so each link is fetched exactly once ever.

That left 342 unnamed. 132 were GitHub urls that hit the unauthenticated rate limit — a token fixes those for free and yields the repo's own description; arXiv and twitter-mirror handlers covered ~20 more. agent-fill.py hands what remains to an agent CLI that can actually read a page, and named 105 of 155.

50 links stay unnamed on purpose. A link that can't be read keeps its url-derived name, because a confident guess about a page nobody fetched is indistinguishable from a real entry and worse than a boring title. Agent-sourced records are marked via so they never masquerade as the page's own metadata, and merging refuses to overwrite a real title.

Result: 1,854 of 1,927 links titled (96%), 1,638 blurbed (85%).

Dedupe

taxonomy.canonical() reduces a url to a dedupe key: strips tracking params, unifies youtu.be with youtube.com/watch, collapses any GitHub url to owner/repo, and treats every spelling of a tweet (x.com, twitter.com, the vx/fx mirrors, /i/status/) as one post — the status id is the identity, since the handle in the path isn't even required to be correct. That last one alone merged 239 duplicates, which existed because a message and its auto-embed spell the host differently.

Addressing params are kept. A false split leaves a visible duplicate; a false merge silently deletes an entry. Testing the naive host+path rule showed it would have merged 35 distinct YouTube videos into one.

One row survives per key; the newsletter copy wins the display since it carries a hand-written blurb, and every other sighting folds into also. The entry's first date becomes the earliest sighting anywhere, which is what makes a row read #100 · first in #reads, Sept 2024.

This also fixes 21 pre-existing duplicates and 5 footnote-anchor entries in the newsletter index, independent of Discord.

On the page

A newsletter entry shows its edition number in the left rail; a Discord-only link shows the Discord mark in that same slot — the rail is an address and this one has no edition to point at.

Time filters (Last week / month / 3 months / year) measure the most recent sighting, so a 2023 link reshared yesterday counts as this week. They ride in the url as #w=7d.

Section, channel and topic fold into one collapsible row, closed by default — 34 channel chips over five lines pushed results most of a screen down. Time range stays out in the open. A filter left on inside a closed card would be invisible, so the summary reads Filtering by #tools, and a deep link into a tag arrives with the card open.

Daily job

.github/workflows/daily-discord.yml at 04:17 UTC: harvest, enrich new links, rebuild, commit. Needs repo secrets DISCORD_TOKEN and DISCORD_GUILD_ID (both set).

Verified by running every step from a fresh --depth 1 clone with only what CI has — no .env.local, no venv, no Substack export. A quiet day now produces no commit at all.

Two bugs that would have broken it unattended:

vectors.f32 was rebuilt on a calendar, weekly, while the index rebuilds daily. Any mismatch makes the page fall back to embedding every entry in-browser, which at 3,471 entries locks the tab for a minute — so a stale vectors file is a broken page, not a slower one. vectors-stale.py (stdlib only, so the check costs no install) now gates the rebuild on the actual row count.
fetch used cache: 'force-cache' on a fixed vectors.f32 url, which prefers a cached copy regardless of freshness. Once the job shipped a new file, returning visitors kept serving themselves the old one and hit exactly that fallback. The row count now rides in the url.

Also: a 45-minute timeout so a wedged fetch fails instead of idling, and rebase-and-retry around the push.

Not in CI, by design

build-index.py needs the Substack export (hence the committed data/nibble.json); agent-fill.py needs an agent CLI and is a manual maintenance pass.

Still open

Payload. vectors.f32 is 5.08 MB and now rebuilds whenever content changes — roughly a gigabyte of git history a year. int8 quantization would cut it 4× to ~1.3 MB and is the same change that answers first-load weight.
#liked-phrases holds quotes rather than links, so it's invisible to a link indexer. Surfacing it needs a separate path that indexes message text.

Testing

make-fixture.py writes a synthetic harvest — cross-source duplicates, short links, tracking params, chat furniture, multi-channel dedupe — so the merge can be tested end to end with no token.

The #nibble channel finds most links before an edition runs them, and shares plenty that never make it in at all. Both now feed one index. Splits the build in two, because the sources refresh on different clocks. The Substack export is a manual download that only exists on a laptop; Discord refreshes daily in CI and has to rebuild the page without it. So build-index.py now writes data/nibble.json (committed) instead of touching the page, and a new build-page.py merges every source and inlines the result. It is pure: same inputs, same page. A Discord link belongs to its message, not to an edition, so "edition" stops being the spine of the page. The left column shows an edition number or a Discord glyph, sorting and the first/last trace run on dates - the one axis both sources share - and the trace is computed from the live hit set, so a tag chip and a free-text search can no longer disagree about a span. Dedupe is the point rather than a detail: taxonomy.canonical() strips tracking params, unifies youtu.be with youtube.com/watch and collapses GitHub urls to owner/repo, but keeps addressing params, so the 35 distinct videos on youtube.com/watch stay 35 entries. A false split shows a duplicate; a false merge silently deletes an entry. One row survives per link, the newsletter copy wins the display for its hand-written blurb, and every other sighting folds into `also` - which is what lets a row read "#100 · first in Discord, Nov 2024". Most Discord messages are a bare url, so enrich-links.py fetches the missing title and blurb (OpenGraph, plus the GitHub, YouTube, Wikipedia and HN APIs) and caches the result by canonical url. Also drops 5 footnote-anchor entries the parser was picking up as links, and folds 21 duplicates that were already in the newsletter: 1862 entries -> 1836. make-fixture.py exercises the whole pipeline without a Discord token.

The two auth schemes differ - a bot token is sent as "Bot <token>", a user token bare - and they are not distinguishable by shape, so the type is declared via DISCORD_TOKEN_TYPE rather than guessed.

Each channel keeps its own cursor, so adding one to DISCORD_CHANNEL_IDS backfills that channel alone and leaves the rest untouched. The harvest gains a `channels` map and every message records the channel it came from; an existing single-channel harvest migrates forward on load. The cursor now advances past every message SEEN rather than every message kept. It was derived from the newest message holding a link, so a tail of link-free chat was re-fetched on every run, forever. Dedupe already worked across sources and now spans channels too - the same link in #tools and #reads is one entry, and the row reads "also in #tools". The channel name becomes the row's label, where a newsletter entry shows its section, and it joins the search haystack so "#reads" is a searchable term. A channel whose name says what it holds overrides the kind heuristic: a link in #reads is a Read even when it points at GitHub. Also drops fetched descriptions that are consent walls, login prompts or "just a moment" interstitials - never a description of anything, and worse than leaving the field empty for the message text to fill.

…n address Setting DISCORD_GUILD_ID replaces the hand-maintained channel list: each run asks the server what it has and reads every channel @everyone can view, plus threads and forum posts. A channel created next month is indexed the next morning. Private channels, private categories and anything the token cannot read are skipped with a note instead of failing the run. On the page, a Discord-only link now shows the Discord mark where a newsletter entry shows its edition number, and each channel is a chip in its own facet row - so the rail stays an address and the channels never crowd the topic chips.

A token pasted into a shell command lands in history and in scrollback. Sourcing a gitignored file keeps it on the machine and out of both.

DISCORD_GUILD_ID replaces the hand-kept channel list: 38 channels and 71 threads found automatically, 1910 messages holding 2259 links. The harvest stores only the commentary around a link, not the message it came in. data/discord.json is committed to a public repo, and publishing a community's chat verbatim is a different thing from indexing the links in it - 146 KB of message text becomes 40 KB, and the 1316 messages that were a bare link now store nothing at all. Account ids are no longer collected; they were never read by anything downstream. HIDE_CHANNELS drops #memes, #liked-phrases and #introductions from the page without dropping them from the harvest, so the choice is reversible without re-fetching.

The window measures the most RECENT sighting, so a 2023 link someone reshared yesterday counts as this week - which is the honest answer to 'what came up lately', and what the existing `last` field already holds. It filters the semantic path as well as the keyword one, and rides in the url as #w=7d so a 'what did we find this month' view is shareable like every other view. The lede also claimed '3743 entries and 1907 more from Discord' - the 3743 already included them.

Enrichment left 342 links unnamed. 132 of those were GitHub urls that hit the unauthenticated API rate limit - a token fixes them for free and gives the repo's own description, which is better than anything inferred. Handlers for arXiv and the vx/fx twitter mirrors covered another 20. That took the gap to 178 without a single model call. agent-fill.py hands what is left to an agent CLI that can actually read a page, and names 105 of 155. The remaining 50 keep their url-derived names: a link that cannot be read stays unnamed, because a confident guess about a page nobody fetched is indistinguishable from a real entry and worse than a boring title. Agent-sourced records are marked `via` so they never masquerade as the page's own metadata, and merging refuses to overwrite a real title. Job postings are out. #job-posts was mostly announcements on twitter rather than board urls, so hiding the channel does what a host filter could not; taxonomy.is_job() catches strays elsewhere, anchored at the start of the path so an article at businessinsider.in/tech/careers/... survives. 3644 entries, vectors regenerated to match.

@handle

Every twitter link was titled 'Post by @handle' with no description - useless in a search index, and outright wrong on a modern /i/status/ url where `i` is a placeholder rather than a handle. fxtwitter's public JSON returns the post itself, so 321 tweets now carry their real text and author. 284 of them gained a description they never had. canonical() also treated x.com and twitter.com as different sites, so the same tweet was indexed twice - 239 duplicates, since a message and its embed often spell the host differently. The status id is the identity: the handle in the path is not even required to be correct, as the vx/fx mirrors demonstrate. Distinct links 2161 -> 1922, blurb coverage 62% -> 85%, 3468 entries.

fetch used cache:'force-cache' on a fixed vectors.f32 url, which prefers a cached copy regardless of freshness. Once the daily job shipped a new file, a returning visitor kept serving themselves the old one, failed the byteLength check, and silently fell back to embedding every entry in-browser. At 3468 entries that locks the tab for a minute or more - reproduced here, and it would be far worse on a phone. The row count now rides in the url, so a stale copy can never satisfy the request. Also: an og:title that is itself a url named nothing (one site puts its canonical url in <title>), and archive.is/imdb urls end in an opaque id, so reading the url produced 'XArCk' and 'Tt5537002' as titles. Both fall back to the host instead.

Copilot

Pull request overview

Adds Discord as a second content source for the existing Nibble index, merging both sources into a single deduped archive that can be rebuilt in CI without requiring a local Substack export.

Changes:

Introduces a Discord harvester (fetch-discord.py) and link enrichment cache (enrich-links.py) to collect and name Discord-shared links.
Splits the build into a pure merge+inline step (build-page.py) and a Substack-only parser outputting data/nibble.json (build-index.py).
Adds a scheduled GitHub Actions workflow to refresh Discord data daily and rebuild vectors weekly.

Reviewed changes

Copilot reviewed 10 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`taxonomy.py`	Centralizes tagging + URL canonicalization used for cross-source dedupe.
`README.md`	Documents the new two-source pipeline, CI job, and Discord setup.
`make-fixture.py`	Creates a synthetic Discord harvest to test the pipeline offline.
`fetch-discord.py`	Discovers readable channels/threads and incrementally harvests link-containing messages.
`enrich-links.py`	Fetches titles/blurbs for harvested links into a committed cache keyed by canonical URL.
`build-page.py`	Merges newsletter + Discord entries, dedupes by canonical key, and inlines into `index.html`.
`build-index.py`	Refactored to emit `data/nibble.json` only (no longer writes `index.html`).
`agent-fill.py`	Batch helper to fill remaining unnamed links via an external agent CLI.
`.gitignore`	Ignores local `.env.local`.
`.github/workflows/daily-discord.yml`	Scheduled workflow to harvest/enrich/rebuild and commit updates.
`.env.local.example`	Example env file for local Discord runs.

Suppressed comments (1)

taxonomy.py:160

Same .git stripping bug as above: re.sub(r'.git$', ...) will remove any trailing ?git sequence, not just a literal .git suffix, producing incorrect canonical keys for some GitHub repos.

            return f"github.com/{parts[0].lower()}/{re.sub(r'.git$', '', parts[1]).lower()}"

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

34 channel chips over five lines pushed the results most of a screen down. Time range stays out in the open since it is the one filter people reach for without being prompted; the rest sits behind a single line, closed by default. A filter left on inside a closed card would be invisible, so the summary reads 'Filtering by #tools' whenever a chip is active, and a deep link into a tag arrives with the card already open.

Today's run proved the problem: three new links took the index to 3471 while vectors.f32 held 3468, so every visitor would have fallen back to embedding 3471 entries in-browser - which locks the tab for a minute. The weekly cadence meant that was the normal state six days out of seven. vectors-stale.py answers 'do these still match' using nothing but the stdlib, so the job can skip the onnxruntime install on the days nothing changed. Also hardened for unattended running: a 45 minute timeout so a wedged fetch fails instead of idling, and a rebase-and-retry around the push so a human commit landing mid-run does not lose the day's harvest.

The script claims to be pure - same inputs, same page - but stamped a fresh generatedAt on every run, so index.html always differed and the daily job's 'nothing new today' branch could never fire. That is a commit every day forever, whether or not Discord had anything to say. The stamp is now kept when nothing else moved, which also makes 'last updated' on the page mean the last time the index actually changed rather than the last time CI woke up.

Records what changed while building this out: the agent pass and its refusal to guess, the enrich flags for when a handler improves, why a stale vectors.f32 is now a broken page rather than a slower one, HIDE_CHANNELS and the job filter, and the two new page controls.

- taxonomy: r'.git$' treated the dot as any character, so a repo named 'legit' keyed as 'le' and could false-merge with an unrelated project. Nothing in the harvest triggers it today, which is exactly why it was worth catching now. - taxonomy: canonical() lowercased the query string, contradicting the rule right above it. Query VALUES are often case-sensitive ids - a VS Code extension, a YouTube playlist, an archive node - and folding them is the false merge this function exists to prevent. 15 links were affected; the path stays folded, as reasoned there already. - enrich-links: arXiv over https, not http. - agent-fill: mkdtemp instead of mktemp, so the agent's output path is not a name another process can win a race for. Cleanup is one rmtree. Cache reconciled: 15 newly-keyed links fetched, 15 orphans pruned.

Harden the Discord ingestion pipeline

Fresh harvest and enrichment on the merged changes: 3476 entries, 1836 from the newsletter and 1640 Discord-only, 1856 of 1932 links titled. Discord input drops 2019 -> 1780 because the per-message key dedupe now folds the embed Discord attaches to a link into the same sighting, rather than counting it as a second one in the same channel. Vectors regenerated against the new content fingerprint.

aashutoshrathi added 8 commits August 27, 2026 19:25

Accept a personal Discord token as well as a bot token

86c217a

The two auth schemes differ - a bot token is sent as "Bot <token>", a user token bare - and they are not distinguishable by shape, so the type is declared via DISCORD_TOKEN_TYPE rather than guessed.

Keep Discord credentials in a gitignored .env.local

c7a4387

A token pasted into a shell command lands in history and in scrollback. Sourcing a gitignored file keeps it on the machine and out of both.

aashutoshrathi marked this pull request as draft August 29, 2026 06:52

aashutoshrathi added 2 commits August 29, 2026 14:08

aashutoshrathi marked this pull request as ready for review August 30, 2026 14:52

aashutoshrathi requested a lite review from Copilot August 30, 2026 14:52

Copilot started reviewing on behalf of aashutoshrathi August 30, 2026 14:52 View session

Copilot AI reviewed Aug 30, 2026

View reviewed changes

Comment thread taxonomy.py Outdated

Comment thread taxonomy.py Outdated

Comment thread enrich-links.py Outdated

Comment thread agent-fill.py Outdated

aashutoshrathi added 4 commits August 30, 2026 21:36

aashutoshrathi requested a review from thepushkarp August 31, 2026 14:00

aashutoshrathi and others added 2 commits August 31, 2026 19:33

fix: harden Discord ingestion pipeline

0785cc2

thepushkarp mentioned this pull request Sep 1, 2026

Harden the Discord ingestion pipeline #9

Merged

aashutoshrathi added 2 commits September 1, 2026 19:06

Merge pull request #9 from The-Nibble/codex/harden-discord-pipeline

53e2a96

Harden the Discord ingestion pipeline

aashutoshrathi merged commit d972f1e into main Sep 3, 2026
2 checks passed

Navigation Menu

Uh oh!

Index Discord links alongside the newsletter, deduped - #8

Index Discord links alongside the newsletter, deduped#8
aashutoshrathi merged 18 commits into
mainfrom
discord-nibbles

aashutoshrathi commented Aug 29, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

aashutoshrathi commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it fits together

Channels

Privacy

Naming the links

Dedupe

On the page

Daily job

Not in CI, by design

Still open

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aashutoshrathi commented Aug 29, 2026 •

edited

Loading