fix(search): sitemap discovery + rollout settle for the index scraper - #1271
Open
ndneighbor wants to merge 1 commit into
Open
fix(search): sitemap discovery + rollout settle for the index scraper#1271ndneighbor wants to merge 1 commit into
ndneighbor wants to merge 1 commit into
Conversation
…efore scraping The scraper discovered pages only by following links from /, so a scrape racing a deploy crawled the stale cached /guides index and missed most of a 38-guide batch (verified against the prod index: llm-gateway present, code-execution-sandboxes/webhooks-at-scale/ uptime-kuma absent, run 30519545306). Discovery now also reads sitemap-0.xml, which regenerates with every build, and deployment-triggered scrapes wait 3 minutes for the rollout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
🚅 Deployed to the docs-pr-1271 environment in 🪄 *.railway.com
6 services not affected by this PR
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Docs search was missing most of the 38 guides merged in #1267. The index scrape (run 30519545306) completed 'successfully' at 06:25 with 33,376 records but only crawled 1 of the new guides.
Root cause: discovery is pure link-crawling from
/, and the scrape fires on the deploy-success event, racing the rollout. It crawled the stale cached/guidesindex (pre-deploy copy with no links to the new guides), so pages only discoverable through fresh links were invisible.llm-gatewaysurvived only because an updated page that happened to serve fresh linked it.Fix
sitemap_urls: the scraper now seeds fromsitemap-0.xml(371 URLs, regenerated by next-sitemap on every build), so new pages are discovered regardless of what the crawl-path caches serve.Verification
A manually dispatched re-scrape (run 30575053481) is in flight against the now-fresh cache; I'll confirm the missing guides are searchable once it lands. This PR prevents the recurrence.
🤖 Generated with Claude Code