fix(llms): serve markdown in place at the docs root - #86
Merged
Conversation
The homepage answered markdown clients with a 307 to /llms.txt, which Vercel serves as text/plain. Agent-readiness scanners follow the redirect, see a non-markdown content type, and score the site as not supporting markdown negotiation. Rewrite to /AGENTS.md instead, so the root returns 200 text/markdown at the requested URL. The guide opens with the /llms.txt index pointer, so agents still reach the full index in one hop. Browsers are unaffected and still land on /overview.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
isitagentready.comscoreschecks.contentAccessibility.markdownNegotiationas fail for docs.steel.dev:The scanner only probes the homepage.
middleware.tsredirected/to/llms.txt, and Vercel servespublic/llms.txtastext/plain, so the final response never carried a markdown content type. Every other negotiation path was already correct (/overview/intro-to-steeland/overview/intro-to-steel.mdboth returntext/markdown; charset=utf-8).Change
Rewrite instead of redirect:
/with markdown intent now serves/AGENTS.mdin place, which returns200 text/markdownand opens with> Full docs index: https://docs.steel.dev/llms.txt. Content negotiation stays on the requested URL, and agents still reach the index in one hop.Verification
Production build (
bun run build && next start):GET /+Accept: text/markdown/llms.txt,text/plaintext/markdown; charset=utf-8GET /from curl/llms.txt,text/plaintext/markdown; charset=utf-8GET /from a browser/overview/overview(unchanged)Tests: 3 new middleware cases plus one e2e case covering the root; full suite 209 pass / 0 fail;
bun run checkclean.Follow-up, not in this PR
Negotiable docs URLs answer with HTML or markdown but their
Varydoes not listAcceptorUser-Agenton prerendered pages. Next's app router callssetVaryHeaderitself and discards values set vianext.config.mjsheaders (verified: a probe header on the same rule applied,Varydid not), so fixing it needs the Vercel routing layer and a preview deploy to verify. Vercel's own cache keys after middleware runs, so the exposure is limited to downstream shared caches.