[finding] llms-full.txt emits HTML numeric entities into a plain-text file, and two of them break the markdown link they sit in

[hotlong]

Found while implementing #178. Pre-existing on main — the counts below are byte-identical before and after that change, which is how it was spotted.

Notation. This tracker's body sanitizer decodes HTML numeric character references, including inside code fences — the first version of this card was rewritten by it into one where the before and after examples read identically. So throughout this card, AMP stands for one literal ampersand character. AMP#x2A; means the six characters that would otherwise be typed as an ampersand, hash, x, 2, A, semicolon. That behaviour is itself worth knowing about when filing evidence here.

What is on main

The body served at /llms-full.txt contains HTML numeric character references, in text that is not HTML:

entity	character it encodes	occurrences
AMP#x2A;	asterisk	58
AMP#x60;	backtick	5
AMP#x29;	closing parenthesis	4

They come from page.data.getText('processed'), via getLLMText in apps/docs/lib/source.ts.

The authored MDX is plain. content/docs/build/index.mdx:36:

1. **Just exploring?** → Open ObjectOS, start the [AI Builder](/docs/build/ai-builder),
   say *"I need to track customer support tickets ..."*

The served text for that same line, with the entities spelled in the notation above:

1. **Just exploring?** → Open ObjectOS, start the [AI Builder](/docs/build/ai-builderAMP#x29;,
   say AMP#x2A;"I need to track customer support tickets ..."*

Two things happened: the closing parenthesis of the link target became AMP#x29;, and the opening asterisk of the emphasis pair became AMP#x2A; while its closing partner stayed a literal asterisk.

Why it is worth a card

Two effects, one cosmetic and one not.

Cosmetic: an LLM reading this file reads AMP#x2A; literally. It is not markdown emphasis and it is not an asterisk; it is six characters of noise inside a sentence, 58 times. The whole file exists to be read as text by something that will not run it through an HTML parser.

Not cosmetic: where the escaped character is a link's closing parenthesis, the markdown link stops being a link. Two of the four AMP#x29; occurrences do exactly that, both on [AI Builder](/docs/build/ai-builder... — one in content/docs/build/index.mdx, one in content/docs/build/ai-builder.

Both render correctly on the website, because there the entity decodes back to a parenthesis before markdown ever sees it. In llms-full.txt nothing decodes it, so the link is delivered as broken literal text.

These are the only 2 of the 646 link targets in the served body that are not well-formed; the other 644 are fine.

Measured on main @ 6653744, production build, fetched over HTTP.

Shape of the fix

Not obvious enough to prescribe, which is why this is a finding rather than a task. getText('processed') is fumadocs' own pipeline and the escaping happens inside it. Candidates: ask for a different text mode; decode the numeric references on the way out in getLLMText; report it upstream. Worth checking which characters fumadocs escapes and why before picking — a blind decode of every numeric reference would also rewrite one an author typed on purpose.

/llms.txt is unaffected: it emits titles and descriptions from frontmatter, not processed body text.

Out of scope for #178, which changes how URLs in these bodies are spelled and leaves the 2 malformed targets exactly as it found them. #178 is not addressed here.

Reproduce

cd apps/docs && pnpm run build
grep -o 'AMP#x[0-9A-Fa-f][0-9A-Fa-f];' .next/server/app/llms-full.txt.body | sort | uniq -c

(with AMP typed as a literal ampersand)

---

Generated by Claude Code

[hotlong]

Graded: promoted to pm:queue / Bug, finding removed.

Filed as a finding because the fix is not obvious — correct call, and it stays a finding-shaped problem in a Bug-shaped card. Two of 646 link targets are delivered malformed, which is a defect with a measurement, not an observation. The other half (58 asterisks and 5 backticks of literal noise in a file whose entire audience will not run it through an HTML parser) is real but would not on its own have earned a Bug.

The first task is investigation, not a patch

The card names three candidates — a different text mode, decoding on the way out in getLLMText, reporting upstream — and warns that a blind decode would also rewrite a reference an author typed on purpose. That warning is the card's most valuable line and it should govern the dispatch.

So whoever takes this establishes first, before writing anything:

1. Which characters getText('processed') escapes, and why. *, ` and ) are all markdown-significant — the escaping looks deliberate, aimed at a consumer that will re-parse the text as markdown. If that is what it is, "decode everything" is fighting the pipeline rather than configuring it.
2. Whether fumadocs offers a text mode that does not escape. Check the installed version's API before assuming it does not.
3. Why the escaping is asymmetric — the card measured an emphasis pair where the opening asterisk became an entity and the closing one stayed literal. That asymmetry is a clue about where in the pipeline it happens, and any fix that does not explain it is probably treating a symptom.

If the answer turns out to be "report upstream and wait", that is a legitimate outcome — say so with the reproduction rather than shipping a local decode that papers over it.

Scope note

/llms.txt is unaffected (frontmatter titles and descriptions, not processed body text), so this is one route's problem. The file has changed twice today — #178 (9a1a7ad, absolute links) and #196 is queued to change its page order — so read the tree, and expect to sequence against #196 since both land in llms-full.txt/route.ts.

The notation finding is worth more than this card

Buried in the body:

This tracker's body sanitizer decodes HTML numeric character references, including inside code fences — the first version of this card was rewritten by it into one where the before and after examples read identically.

That is the second thing we have learned about this sanitizer today; the first was that it strips angle-bracket markup, including inside fences. A card whose entire subject is character-level escaping was silently rewritten into one that showed no difference between the broken and correct forms — the failure mode being documented, applied to the document describing it.

Inventing the AMP notation to route around it, and stating the convention up front, is the right handling. Recording it here so the next agent filing evidence about escaped characters in this tracker does not lose an hour to it.

---

Generated by Claude Code

[hotlong]

Claim: PM loop round 9
Session: session_01G5zjYc2BoFV2NjKBBapC7C
Branch: claude/issue-197-llms-full-entity-decode
Worktree: objectos-issue-197
Domain: n/a (objectos carries no domain:* lanes)
File surface: apps/docs/app/llms-full.txt/route.ts, apps/docs/lib/source.ts (getLLMText lives there). Stop on breach; explain in the report.
Container & model: M, mode:subagent, model: opus — PM judgement.
Clause-②: no
Serial constraints cleared: llms-full.txt/route.ts is free — #178 (9a1a7ad) and #196 (1ebd7b7) have both merged. #184 is in flight on .github/scripts/check-locale-surface.mjs and #199 on app/layout.tsx / middleware.ts / lib/seo.ts; neither touches your files.

Read coupling with #184, and it is why the scope line matters: #184 is adding this file's body to the locale-surface gate, asserting which pages and which locales appear. Your change alters the characters inside page text. Those are disjoint — page titles come from frontmatter and never pass through getText('processed') — so the gate will not notice you. But if your fix changes which pages appear, stop and report: that would collide.

Note the file has been rewritten twice in the last two hours. Read the tree.

---

Generated by Claude Code

[hotlong]

Claim (dev): starting investigation.

Session: session_01G5zjYc2BoFV2NjKBBapC7C
Branch: claude/issue-197-llms-full-entity-decode (pushed, empty, at 1ebd7b7)
Worktree: /home/user/objectos-issue-197
Base: 1ebd7b7 — origin/main after #196/#201 landed, so both of today's rewrites of this file are in the tree I am reading.
File surface: apps/docs/app/llms-full.txt/route.ts, apps/docs/lib/source.ts. Stop on breach and report.

Order of work is taken from the grading comment: investigation first, three questions answered before any patch — which characters getText('processed') escapes and why, whether the installed fumadocs has a non-escaping text mode, and what explains the asymmetry the card measured. needs_decision with a reproduction is on the table as a terminal outcome; a blind decode is not.

Following the card's AMP notation for anything I write back here.

---

Generated by Claude Code

[hotlong]

os-dev-report

{
  "issue": 197,
  "status": "needs_decision",
  "branch": "claude/issue-197-llms-full-entity-decode",
  "pr": null,
  "premise_still_valid": true,
  "files_changed": [],
  "gates": "None run: the diff is empty, so there is nothing for a gate to judge. Two full production builds of apps/docs were run instead (baseline and candidate-fix), both through the shared verify lock, both VERDICT command-exit 0.",
  "deviations": [
    "No PR opened. The only repair that addresses the cause needs pnpm.patchedDependencies plus a patches/ file, which is outside the declared file surface (apps/docs/app/llms-full.txt/route.ts, apps/docs/lib/source.ts). Stopped on breach and reported, as instructed.",
    "The candidate fix was proved by patching node_modules, never the repo. The target file is hard-linked into the pnpm store (link count 3), so the directory entry was replaced with mv rather than written through the shared inode (link count 1 after, confirming the store inode was untouched); an EXIT/INT/TERM trap restored it and the restore was verified by sha256 equality with the pre-experiment hash e061d718f43d9ec08ef0e52b41c7a5b8cde84512b94fe133710c90039a8a3246.",
    "Branch claude/issue-197-llms-full-entity-decode was pushed empty as the write-route probe and is left in place at 1ebd7b7 (identical to origin/main). It carries no commits; delete it or reuse it, whichever suits the route chosen."
  ],
  "summary": "NOTATION: AMP stands for one literal ampersand character, per the card's convention, because this tracker's sanitizer decodes numeric character references even inside fences. Investigation only; nothing shipped. The card's premise holds exactly: on origin/main at 1ebd7b7 the served body carries 58 AMP#x2A;, 5 AMP#x60; and 4 AMP#x29; -- the card's counts to the digit. ANSWER 1 (what getText('processed') escapes, and why). Its escaping mechanism is the BACKSLASH, not the character reference. The same served body carries 51 backslash escapes across 7 distinct characters (19 underscore, 15 pipe, 6 backtick, 5 tilde, 3 left angle bracket, 2 ampersand, 1 asterisk). That escaping is deliberate and is aimed at exactly the consumer the grading comment guessed at -- something that will re-parse the text as markdown. The 67 numeric references are NOT that policy. They come from a different function, encodeCharacterReference, reached only through mdast-util-to-markdown's attention-adjacency rule, whose own source comment reads: 'Important: punctuation must never be encoded. Punctuation is solely used by markdown constructs. And by encoding itself. Encoding them will break constructs or double encode things.' The one place a character reference is legitimate for punctuation is a fenced code block's info string (safe() is called there with encode set to backtick); none of the 67 sit in one. The file spells the SAME characters both ways -- asterisk appears as backslash-asterisk and as AMP#x2A;, backtick as backslash-backtick and as AMP#x60; -- which on its own proves the two are not one policy. So 'decode everything' is not fighting the pipeline: all 67 are outside what the pipeline intends, and the pipeline's own intended spelling for them is the backslash. ANSWER 2 (does the installed fumadocs offer a non-escaping text mode). No. Read from node_modules, not from docs: fumadocs-mdx 15.0.7 exposes exactly two modes, getText('raw') (reads the original file off disk -- frontmatter, imports and JSX components included, so not a substitute) and getText('processed'). includeProcessedMarkdown accepts LLMsOptions, which is mdast-util-to-markdown Options plus fumadocs' element filters; there is no escaping switch on it, and any handlers passed through it are wrapped by the very code that causes the defect, so that channel cannot even be used to work around it. There is no configuration route. ANSWER 3 (the asymmetry, and the root cause it points at). fumadocs-core's defaultStringifier wraps every serializer handler (modHandler, dist/mdx-plugins/stringifier.js) and does not carry the handler's .peek property onto the wrapper. mdast-util-to-markdown's containerPhrasing uses handle.peek to obtain the NEXT sibling's first character cheaply and WITHOUT side effects; with peek missing it invokes the next sibling's FULL handler instead, and that lookahead call leaves state.attentionEncodeSurroundingInfo set from node N+1. The loop then reads those flags as though they described node N and applies them to (a) the last character ALREADY EMITTED for sibling N-1 and (b) the first character of sibling N+1. That is the asymmetry, precisely: the emphasis run's OPENING marker is encoded because it is the first character of the following sibling, while its CLOSING marker is left alone because when that node is finally serialized for real its own encodeInfo has true context and correctly asks for no encoding. It also predicts every damaged character in the file, and the prediction holds: each one is either the closing delimiter of a peek-bearing node -- strong (asterisk), link (right parenthesis), inlineCode (backtick) -- or the opening delimiter of an attention run. 58 + 4 + 5 = 67, with nothing left over. THREE CORRECTIONS TO THE CARD, all of which make it worse rather than better. (a) FOUR link targets are malformed, not two, out of 650, not 646. The card's 646 looks like its well-formed count used as the denominator: 646 well-formed + 4 broken = 650. The four are content/docs/build/index.mdx:36, content/docs/build/interface/dashboards.mdx:214 (the card said 'content/docs/build/ai-builder'), content/docs/configure/mcp.mdx:7 and content/docs/resources/faq.mdx:212. All four are ordinary authored links in plain MDX; in every case the trigger is the attention run immediately following. (b) This is NOT one route's problem. /llms.mdx/docs/[[...slug]] prerenders 79 per-page bodies and they carry the identical 58/5/4, because both routes go through getLLMText and therefore through getText('processed'). The grading comment's scope note is right that /llms.txt is unaffected (frontmatter only) but /llms.mdx is affected. A fix in llms-full.txt/route.ts would repair one route; a fix in lib/source.ts would repair both. (c) The defect is quietly defeating a fix that already shipped. SITE_LINK in llms-full.txt/route.ts needs a literal right parenthesis to match, so the two broken SITE links were invisible to #178's absolute-URL rewrite and are served RELATIVE in a file whose whole purpose is to be read elsewhere. Repairing the entity bug raises absolute site targets from 549 to 551 with no other link touched. WHY NOTHING SHIPPED. The cause is one missing line in fumadocs-core 16.8.12, and it is still present in 16.15.1, the current latest -- that function is byte-identical to the installed one apart from a brace reformat -- so 'upgrade' is not a fix and 'report upstream and wait' has no bounded end. Repairing the cause from this repo means pinning the upstream one-liner with pnpm.patchedDependencies, which is outside the declared file surface. The only in-surface repair is an output decode, and the card's warning about it is real and now demonstrated rather than asserted: an author-typed numeric reference in PROSE is decoded at parse time and re-serialized as a backslash escape, so it can never reach the output and a decoder could not hurt it -- but inside inline code and fenced code it survives VERBATIM, and a decode would rewrite it. Worse, the bug itself destroys inline-code closing delimiters (the served body contains backtick-thank-you-AMP#x60;), so a 'skip code spans' guard would be keying on structure the bug has already broken. content/docs carries 0 author-typed numeric references in English MDX today, so that hazard is prospective rather than present -- but it would be a permanent output-rewriting step in this repo, guarding against a bug we could instead delete.",
  "tests": "All measurements on worktree /home/user/objectos-issue-197 at BASE 1ebd7b7 (origin/main, after #178/9a1a7ad and #196/1ebd7b7 both landed). Heavy commands via bash /home/user/objectstack/scripts/pm/os-verify-lock.sh -c '...'. (1) BASELINE BUILD. 'cd apps/docs && NODE_OPTIONS=--max-old-space-size=4096 pnpm run build' -- 'os-verify-lock: VERDICT command-exit 0 . held the lock 57s . waited 0s'. Card's reproduction on .next/server/app/llms-full.txt.body: 58 AMP#x2A;, 5 AMP#x60;, 4 AMP#x29;, 0 decimal references -- identical to the card, so premise_still_valid true. Link targets: 650 occurrences of right-bracket-left-paren, 646 well-formed, 4 malformed. Backslash escapes in the same body: 51 across 7 characters. Same three counts (58/5/4) in all 79 prerendered /llms.mdx/docs bodies. Authored numeric references in English content/docs MDX: 0 files. (2) ISOLATED REPRODUCTION, no MDX and no Next in the loop: fromMarkdown + micromark gfm, then fumadocs-core's own defaultStringifier called with a stub processor supplying gfmToMarkdown. Emits all three entity kinds on a 12-line input -- 'Talk to the AI Builder*AMP#x2A;', '(/docs/build/ai-builderAMP#x29;,', 'backtick-thank-you-AMP#x60; AMP#x2A;(default)*'. This pins the defect to the stringifier, not to MDX, not to fumadocs-mdx, not to this repo. (3) A/B ON THE HYPOTHESIS. Same function re-declared verbatim with one line added -- copy handler.peek onto the wrapper -- run over the same input twice in one process: PRESERVE_PEEK=false gives refs [AMP#x2A;, AMP#x2A;, AMP#x29;, AMP#x2A;, AMP#x60;, AMP#x2A;]; PRESERVE_PEEK=true gives refs []. Output otherwise identical apart from table column widths. (4) END-TO-END ON THE REAL SITE. Mutation: the anchor line 'for (const k in handlers) handlers[k] = modHandler(handlers[k], node.ctx);' occurs exactly once in fumadocs-core/dist/mdx-plugins/stringifier.js; replaced with a form that copies .peek. MUTATION CONFIRMED ON DISK BEFORE THE BUILD, not by an editor's exit code: grep -c of the removed anchor text = 0 and grep -c of the injected marker '__w.peek = __h.peek' = 1, read back from the installed path. The file is hard-linked into the pnpm store (stat link count 3), so the directory entry was replaced with mv rather than written through the shared inode; link count 1 afterwards confirms the store inode was never touched, i.e. no sibling worktree and no future install was poisoned. .next removed so nothing was served from cache, then rebuilt: 'os-verify-lock: VERDICT command-exit 0 . held the lock 57s . waited 0s'. RESTORE LEG: an EXIT/INT/TERM trap invoking an absolute-path restore script ran and printed 'RESTORE OK'; independently re-verified afterwards, sha256 of the installed file = e061d718f43d9ec08ef0e52b41c7a5b8cde84512b94fe133710c90039a8a3246 = the pre-experiment hash, and git status in the worktree is empty with an empty diff against BASE. (5) RESULTS OF THE CANDIDATE FIX, measured on the rebuilt body. Numeric references 67 -> 0. Link targets 650 total in both; well-formed 646 -> 650; malformed 4 -> 0. Membership and order from #196 UNTOUCHED: the sequence of every top-level heading line is byte-identical between the two bodies (93 headings, same order), and the line count is identical (12835). Otherwise byte-identical apart from the decoded characters, checked by decoding the baseline and diffing: 21 differing lines, of which 19 are GFM table column padding (a cell that loses 5 characters narrows its whole column, so the padding of every row is recomputed) and exactly 2 are the AI Builder links acquiring the absolute URL that #178 had been unable to give them -- absolute site targets 549 -> 551. (6) AUTHOR-TYPED REFERENCE PROBE, which is what makes the card's warning concrete. Through the same stringifier: in prose, an authored reference is decoded at parse time and comes back out as a backslash escape (never as an entity); inside inline code and inside a fenced block it survives verbatim, 4 of 4 in the probe. So a decode is safe against today's content and unsafe against a page that documents character references.",
  "open_questions": [
    {
      "question": "The cause is a one-line omission in fumadocs-core (modHandler in dist/mdx-plugins/stringifier.js does not carry Handle.peek onto the wrapper), it is present in the current latest release 16.15.1 as well as the installed 16.8.12, and the fix for it is outside the file surface I was given. Which route should this repo take?",
      "options": [
        "A -- Pin the upstream one-liner with pnpm.patchedDependencies (a patches/ file plus a pnpm-workspace.yaml entry) and file it upstream in parallel. Fixes the cause once, for llms-full.txt and llms.mdx and any future consumer of getText('processed'). Verified end-to-end above: 0 references, 650/650 link targets, membership and order untouched. Cost: two files outside my surface, and the pin must be re-checked on each fumadocs bump (it deletes itself on the release that carries the fix).",
        "B -- Targeted decode inside getLLMText in lib/source.ts (in surface), restricted to numeric references whose code point is ASCII punctuation -- justifiable character by character, because the serializer's own invariant is that punctuation is never encoded, and the only legitimate punctuation reference (a backtick in a fenced block's info string) does not occur here. Cost: a permanent output-rewriting step; it needs a code-span and fence guard, and the bug itself destroys inline-code closing delimiters, so that guard keys on structure the bug has already broken; and it leaves the cause in place for anything that reads getText('processed') without going through getLLMText.",
        "C -- Report upstream and wait, changing nothing here. Cost: unbounded, because the defect is in the newest release, not only in the pinned one. Four link targets stay broken and two stay relative in the meantime."
      ],
      "recommendation": "A, with the upstream report filed at the same time; do not decode. Real business need: measured, not speculative -- 4 of 650 link targets malformed in a file whose entire audience is a machine that will not run it through an HTML parser, plus 2 links silently exempted from the absolute-URL rewrite that #178 shipped for this very file, plus 58 asterisks and 5 backticks of literal noise, and all of it duplicated across the 79 per-page /llms.mdx bodies. Long-term soundness: the contract-first rule says a defect that tempts you to widen the consumer belongs in the producer, and B is exactly the tolerant-consumer shape that rule names; A is byte-identical to what upstream will ship and disappears on the release that carries it, whereas B is code this repo owns forever. Making AI-written material hard to get wrong: this file is read by machines, and A restores well-formed markdown at the source so no downstream consumer has to know the quirk exists, while B leaves a decode step that every future reader of getText('processed') has to remember to replicate -- and the one place B could go wrong is a page documenting character references, which is a page a docs site is unusually likely to acquire. Startup scope discipline: A is one pinned line and no new logic; B is new permanent logic plus a guard whose correctness is genuinely hard to argue. A is less code, not more. If the pin is judged too much machinery for two broken links, C is defensible and B is not -- B costs more than A and fixes less."
    }
  ],
  "out_of_scope_findings": []
}

---

Generated by Claude Code