feat: add DOM-structure page-template classification (--templates)#230
Merged
Conversation
Add an opt-in analyze phase that groups internal HTML pages into templates by DOM-structure similarity, using @d-zero/page-cluster. It runs as a dedicated core phase rather than an AnalyzePlugin, since it needs a corpus-wide batch computation that the per-page eachPage/eachUrl plugin model cannot express. - collect-page-stylesheet-urls.ts resolves each page's referenced CSS via resource_ref_edges, chunked and interned for large archives - create-page-cluster-factory.ts builds the PageFactory that resolvePageClusterKeys reads the corpus through, respecting its sequential/multi-pass/same-order contract - classify-page-templates.ts calls resolvePageClusterKeys and caches the result keyed by the archive's path, size, mtime, and page count - nitpicker.ts runs classification once globally after every getPagesWithRefs batch has been accumulated, and merges into the existing analysis/report instead of overwriting it when no plugins ran, so a standalone run never wipes a previous run's columns Results surface as a templateKey column through the existing Table/ analysis-report mechanism, so they flow into Google Sheets reports the same way other analyze plugins' columns do.
Wire AnalyzeOptions.classifyTemplates through to the analyze command as an opt-in --templates flag, independent of plugin selection. --templates alone (no --all/--plugin, and no configured plugins) no longer trips the "no analyze plugins found" / "no valid plugins to run" guards, since template classification is a valid reason to run analyze() with zero plugins selected.
…ot a read-model column
…ation Adds a dedicated page_templates table (page_id PK, WITHOUT ROWID) and a replacePageTemplates write path, mirroring replaceAnalysisViolations's whole-table-replace shape. Unlike violations, a page whose URL can't be resolved back to content_items is silently skipped rather than treated as a hard failure, since losing one page's classification shouldn't discard a multi-thousand-page run.
…ble/report Now that page_templates is a dedicated SQL table (crawler), --templates classification results are persisted only via Archive.replacePageTemplates and no longer merged into analysis/report or analysis/table. This removes the double-write that risked the two copies silently diverging.
… filtering Joins page_templates (by page_id) into listPages, listPagesByTag, listPagesByJsonLdType, joinViewerPageIdsToListItems, and getPageDetail, so --templates classification surfaces as templateKey without a viewer_pages read-model schema change. Every join is gated on hasPageTemplatesTable: an archive that predates --templates, or a read-only connection that skipped schema self-heal, falls back to a NULL literal instead of throwing "no such table". templateKey is also a first-class ListPagesOptions/ListViewerPagesOptions filter. The fast path resolves it via a WHERE page_id IN (subquery) against page_templates rather than a JOIN, since readKeysetWindow's shared SELECT page_id, ... would otherwise make that column ambiguous once a second page_id-bearing table entered the FROM clause.
Adds a Template column to the Pages table alongside a radio filter backed by the archive's distinct template keys. templateKey is a first-class fast-path filter (query's page_templates whereIn-subquery join), so selecting it never forces the legacy write-model fallback the way urlPattern/directory/ header-presence filters do.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
analyze --templates), using@d-zero/page-clusterto group internal HTML pages by structural similarity. This runs as a dedicated corpus-wide phase incore'sNitpicker.analyze(), separate from the per-pageAnalyzePluginsystem, since template clustering needs the whole page set at once.page_templatesSQL table (crawler), written viaArchive.replacePageTemplates— a dedicated table rather than aTable/analysis/reportblob, so the result has one source of truth.templateKeythrough the existing Pages list/detail query surface (query) and viewer UI (viewer): a new column on the Pages table, plus a radio filter backed by the archive's distinct template keys. No new routes, screens, MCP tools, or CLI subcommands — scoped intentionally to "add a column to what already exists."templateKeyfiltering is a first-class fast-path (viewer_pagesread-model) filter: it resolves via aWHERE page_id IN (subquery)against the narrowpage_templatestable, so selecting it does not force the legacy write-model fallback the wayurlPattern/directory/header-presence filters do.page_templatesjoin/select is gated onhasPageTemplatesTable: archives that predate--templates, or a read-only viewer connection that skipped schema self-heal, degrade totemplateKey: nullinstead of throwing "no such table".Why
@d-zero/page-clusterwas purpose-built for this classification; see https://github.com/d-zero-dev/tools/tree/dev/packages/%40d-zero/page-cluster. Real-archive verification (a ~1,400-page site, 63 distinct template groups) surfaced two gaps this PR also closes:templateKeyinto theTable/analysis/reportJSON blob — invisible to the viewer, and a second write path once a dedicated SQL table exists risks the two copies diverging.templateKeyis now written topage_templatesonly.page_templatestable doesn't exist (predates this feature, or a stub connection that skips schema self-heal) would throwno such table: page_templateson any Pages request that joins it. Every query path now checkshasPageTemplatesTablefirst and falls back to aNULLliteral.Test plan
yarn build/yarn lint(0 errors) /yarn test(484 files / 3,415 tests, all green)replace-page-templates.spec.ts(write path — resolve+persist, full replace, empty-map clear, skip-on-unresolvable-URL),read-viewer-page-facets.spec.ts/list-viewer-pages.spec.ts/list-pages.spec.ts/get-page-detail.spec.ts/join-viewer-page-ids-to-list-items.spec.ts(templateKey exposure, missing-table fallback),register-pages-route.spec.ts(templateKey filter stays on the fast path — opaque keyset cursor, not the legacy decimal-offset cursor)analyze --templates, confirmedpage_templatesrows viasqlite3, opened the viewer, confirmed the Template column and filter both render real classification data and respond in single-digit milliseconds via the fast path