feat(page-cluster)!: report cluster selection reasons via onClusterReason

[YusukeHirao]

Summary

• Add onClusterReason, a callback fired once per final cluster (not per page) with a structured ClusterReason: the blocking evidence that grouped it (shared stylesheet set or URL path prefix), the shared DOM-structural token core, per-landmark-type (header/footer/nav/aside/form/search) commonality within the cluster, and the sibling cluster keys it was split from within the same blocking group.
• This directly answers "why are these pages in the same cluster" (e.g. "header/footer are common chrome, but they differ in whether a sidebar nav is present") without page-cluster ever producing human-readable text — callers assemble their own wording from the structured data.
• Breaking change: removes includeLandmarkPositions, PageClusterKeyResult, and PageLandmarkReport (never used in production). That design held every member page's landmark data until final clustering completed, which is why it rejected corpora over 20,000 pages outright — a real problem for a 170k-page production crawl. ClusterReason is sized by cluster count instead of page count, so it has no such ceiling and works unmodified on the streaming path.
• Landmark position extraction and chrome/content classification are decomposed into independently-callable, stateless public APIs (extractLandmarks, plus the newly public isChromeLandmarkInstance and jaccardSimilarity subpaths) that callers combine with ClusterReason.landmarks[type].shellTokens themselves.
• CLI: --include-landmark-positions is replaced by --cluster-reasons-file <path>, which writes a clusterKey-keyed ClusterReason object once per run (not per-page JSONL fields) — no page-count limit.

Why

Nitpicker needs to show why pages were grouped into a template cluster, not just an opaque clusterKey. Building this surfaced that the existing landmark-report design conflated two concerns — page-level data (landmark position, needs per-page HTML) and cluster-level data (chrome/shell commonality, needs only cluster-count-bounded aggregates) — and the conflation was the actual cause of the prior 20,000-page ceiling.

Test plan

• yarn build — all 28 packages build cleanly
• yarn lint — clean
• yarn test — 1668 tests pass, including new coverage for:
  • blocking reason variants (css / path / orphanMerge)
  • per-landmark-type commonality (presenceRate / chromeRate / shellTokens)
  • sibling-cluster computation for a real Stage A split within one block
  • streaming path (20,001 synthetic pages) completing without error and firing onClusterReason once per final cluster
  • CLI --cluster-reasons-file success and failure (write error reported as clean exit 1, not an unhandled rejection)
• Verified against a real ~1,400-page production crawl archive: independently recomputed presenceRate/chromeRate for every (cluster, landmark type) pair using only the public API and diffed against ClusterReason's own numbers — matched exactly (the one apparent mismatch traced to the verification script itself, not the library; fixed by clarifying chromeRate's JSDoc about its per-page dedup denominator)

🤖 Generated with Claude Code