Release/v1.2.0 - #23
Merged
Merged
Conversation
…rawler/pipeline package, and bullmqFetchQueue, crawlRegistry, domainCooldownGate, and nodeSampleCache into @zod-crawler/pipeline-node
…, plus a Chromium vitest project to test it
…limit and a Durable Object daily request cap
… a ui/general/crawl split and Node subpath imports for a tsc -b composite build
…ets and redirecting everything else on zodcrawler.figulus.dev to the repo, and switch cors-proxy from a Custom Domain to a plain Route on the same zone
…awl UI from @zod-crawler/components with local-only state and a stubbed crawl submit
…apps/web-demo's Start crawl to a real Web Worker running fetchAndCacheSamples, inferSchema, validateSamples, and prettier/standalone
…r/web-demo for Vercel deployment
…everse proxy in tools/site-router, and a missing vite base path fix for apps/web-demo
…split, apps/web-demo, @zod-crawler/components, and the cors-proxy/site-router tools
…nts, wired into apps/web-demo above the crawl UI
Feat/browser pipeline
… --ids/--ids-file and dropping the redundant flags: {} in sync-versions
migrate from zod-cli-flags to zod-commands
update to zod-commands v1.1.2 from v1.1.1
… for its per-command override schema support
add a status subcommand for --detach jobs
…and apps/web's Dockerfiles, and @zod-crawler/components to apps/web's, so both images build again
…es/core READMEs, and the web-demo browser-only banner, broken by tools/site-router serving docs under /docs instead of the domain root
…s site, with a new docs/demo.md page, a docs/_sidebar.md entry, and an apps/web-demo README
…:browser, and correct what build/typecheck actually run per-workspace
Fix/release readiness
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[1.2.0] - 2026-08-15
Added
@zod-crawler/pipeline: a new package, split out of@zod-crawler/core, holding the platform-portable half of the crawl pipeline (fetchAndCacheSamples, theSampleCacheinterface,hashId,hasCachedSample,fetchProgressEvent) with zero Node dependency of its own.@zod-crawler/pipeline-node: a new package holding the Node/Redis backend (createNodeSampleCache,runBullmqFetchQueue,domainCooldownGate,redisConnection,crawlRegistry), depending on@zod-crawler/pipeline.apps/cliandapps/web's server now branch onredisUrlthemselves to pickfetchAndCacheSamplesorrunBullmqFetchQueue, instead of that branch living inside a single fetch function.@zod-crawler/pipeline-browser: a new package withcreateCacheApiSampleCache, a browserSampleCachebacked by the Cache API, andcreateBrowserUrlFetcher, afetch-based fetcher that retries through a CORS proxy only when a direct request throws (network/CORS failure), remembering per-job which origins already needed the proxy.apps/web-demo: a new client-only Vite/React app rendering the sameCrawlForm/JobList/JobDetailasapps/web, running a full crawl (fetchAndCacheSamples->inferSchema->validateSamples->findCrawlCandidates) entirely in a browser Web Worker with no server, using@zod-crawler/pipeline-browser's cache and fetcher andprettier/standalonefor formatting. Deployed at https://zodcrawler.figulus.dev/demo. A crawl doesn't survive a page refresh (no SSE/job registry, unlikeapps/web) - a refresh just restarts.@zod-crawler/components: a new shared shadcn/Tailwind UI package holding the UI and crawl components moved out ofapps/web, soapps/webandapps/web-demorender from the same components.tools/cors-proxy: a self-hosted CORS proxy atzodcrawler.figulus.dev/proxythat the demo falls back to when a target API doesn't allow direct cross-origin fetches, rate-limited per IP and capped daily.tools/site-router: routeszodcrawler.figulus.dev's docs, the new demo, and everything else to the right place.@zod-crawler/cli: astatussubcommand (zod-crawler status --output <dir>) reports on a crawl started with--detach- running, finished (with its recorded exit code), or crashed - by reading<output>/zod-crawler.pidand<output>/zod-crawler.status.json, both written by the detached run itself.Changed
ci.yml) andapps/cli/apps/web's Docker images now run on Node 22/24/26 instead of 20/22/24.@zod-crawler/cli: upgraded fromzod-cli-flags@1.0.2to its renamed successor,zod-commands@1.1.3.--ids/--ids-filenow useexclusiveGroupsto declare "exactly one of" instead of a hand-written check in the parse transform, socli.usagerenders it as(--ids <value> | --ids-file <value>).