add custom Docusaurus plugin generating llms(-full).txt - #449
Conversation
saroupille
left a comment
There was a problem hiding this comment.
Read the plugin end to end — the approach is sound and the risk to the site is nil (postBuild-only: it writes two extra files at the end of the build; a bug breaks the build, not the content). Approving so it can be merged on Thursday as-is. Three non-blocking notes:
-
slug:frontmatter isn't honored.fileToUrlassumes path = URL. Concretely,docs/overview/index.mdhasslug: /, sollms.txtwill link the homepage ashttps://docs.etherlink.com/overview— which 404s on the live site (verified). It's the only affected page today (tutorials/predictionMarketandevm/bridging/bridgingoverrides happen to coincide with their paths). Easy fix: parsesluginextractFrontmatterand use it for the printed URL, while keeping the path-derived key for sidebar matching. -
Pages missing from
sidebars.jsare silently dropped from both outputs (if (!doc) continuecovers the reverse case, but a doc file absent from the sidebar just vanishes). Aconsole.warnlisting orphaned docs would catch drift as pages get added later. -
Raw JSX remains in
llms-full.txtfor.mdxpages (<InlineCopy>,<Link>tables…) — imports are stripped but components aren't. Acceptable noise for LLM consumption; just noting it's known.
🤖 Review researched with Claude Code on behalf of @saroupille
NB: replaces #448 (closed)
Solves issue #438
That is, adds a custom plugin for Docusaurus that scans the doc sidebar and generates standard files for ingestion by agents, called llms.txt and llms-full.txt.
The custom plugin is justified by the fact that it does some specific cleanup actions, such as removing the markers that we use for rebranding the doc back and forth.