PRD: Deliver the maintained HTML-to-Algolia extraction pipeline

[aileen]

Problem Statement

Ghost publishers using the public Ghost-to-Algolia tools depend on an abandoned HTML extraction dependency and a set of coupled legacy behaviors that are difficult to maintain safely. The current system can preserve unwanted HTML-search behavior, produce records that exceed Algolia's strictest public-plan size limit, omit useful Ghost fields, rewrite user-owned index settings during webhook updates, index only posts, and use mutable slugs as operational identity. Replacing one component without an explicit compatibility and migration route could silently change search results, leave stale records behind, collide posts and pages, expose live editorial data in tests, or require an unsafe coordinated release.

The completed Wayfinder map in #186 resolves the product, compatibility, API, evidence, migration, and release decisions needed to replace that pipeline. What remains is to deliver those decisions in dependency order through independently reviewable changes and verified public-package releases, while keeping every existing consumer on a recoverable path.

Solution

Deliver a maintained, strict-TypeScript HTML-to-Algolia pipeline in staged public-package releases. Introduce a narrow @tryghost/algolia-html-extractor package that turns rendered HTML into ordered extraction fragments, first with exact legacy final-record compatibility and later with evidence-backed richer rendered meaning. Add a deep fragmenter record-building interface that owns Ghost content projection, deterministic complete-record size handling, and stable errors while retaining deprecated compatibility wrappers. Update the CLI and Netlify adapters only after the required library releases are publicly verified, stop incremental webhooks from changing index settings, and add explicit post/page selection and Ghost content identity through a guarded one-time index migration.

Use immutable Ghost-rendered fixtures for deterministic offline pull-request checks and a separate aggregate-only live smoke workflow for structural drift. Release each behavior-changing package in the finalized order, verify its packed public seams and npm artifact before activating it in consumers, and preserve documented rollback boundaries throughout.

GitHub Issues Addressed

Issue	How this implementation addresses it	Delivery and closure gate
#23 — Searchable attributes in the config file are not added to algolia	The originally reported CLI defect is no longer reproducible because the CLI already passes configured index settings. The remaining related defect is that Netlify publish events reapply package defaults. The first implementation slice stops all webhook settings reads and writes, retains explicit CLI/indexer setup, documents the required operational facet, and allows a narrower mutation-only runtime key.	Close only after the Netlify patch is published and a clean packed-handler verification proves that publish events make no settings request.
#43 — Extend algoliaPost with additional ghost post field	The deep fragmenter API adds one shared Ghost content projection with an explicit allowlist, validated aliases, deterministic values and errors, and identical CLI/Netlify adapters. Deprecated compatibility wrappers remain unchanged.	Close only after the fragmenter API and both CLI and Netlify projection releases are published and their packed public seams are verified.
#148 — html tags in search results	The default Ghost content projection adds Ghost computed excerpt as display text in every Algolia record while retaining fragment html for search and rich rendering. Projection does not automatically rewrite searchable, retrieval, snippet, or display settings.	Close only after the CLI and Netlify projection releases are published and verified. Closure notes must explain that consumers choose excerpt in their result UI and own any corresponding Algolia settings.
#4 — Cannot index pages instead of posts	The CLI adds validated contentTypes selection for posts, pages, or both; independently paginates both Ghost content types; introduces stable Ghost content identity and complete-run reconciliation; and adds identity-aware page publish/unpublish handlers after the migration gate.	Close only after the CLI page-selection and Netlify page-synchronization releases are on npm, the one-time identity migration/settings contract is verified, and packed/native handler acceptance proves posts-only compatibility plus page synchronization.

The closed Wayfinder tickets linked from #186 are decision inputs to this PRD. They are not additional product defects being claimed as fixed.

User Stories

1. As a Ghost publisher, I want the search indexing pipeline to use a maintained HTML extractor, so that my search integration is supportable on current Node and Ghost versions.
2. As an existing Ghost Algolia user, I want the extractor replacement to preserve my current final Algolia records exactly, so that a maintenance upgrade does not silently change search results.
3. As an existing package consumer, I want legacy fragmenter entry points to remain available during migration, so that I can upgrade dependencies before adopting new record behavior.
4. As a CommonJS consumer, I want to call the extractor synchronously from a packed package, so that I do not need to redesign my current loading path.
5. As an ESM consumer, I want the same named extractor operation and declarations, so that module format does not change the package contract.
6. As a TypeScript consumer, I want stable public declarations that do not expose parser implementation types, so that parser upgrades do not leak into my application.
7. As a package maintainer, I want rendered HTML converted into domain-clean ordered extraction fragments, so that parsing remains separate from Ghost projection and Algolia record policy.
8. As a package maintainer, I want the compatibility release judged at the final Algolia record seam, so that every observable legacy traversal, grouping, ranking, deep-link, and ID behavior is protected.
9. As a search user, I want author-controlled rendered meaning such as captions, table headers, quotations, citations, and meaningful image alternatives to become searchable in the richer release, so that relevant content is discoverable.
10. As a search user, I do not want generated controls, provider chrome, decorative content, URLs, or hidden non-semantic content indexed as meaning, so that results remain useful.
11. As a Ghost publisher, I want richer extraction to preserve the legacy compatibility stream, so that new coverage does not alter established fragments unexpectedly.
12. As a package consumer, I want every extraction fragment to carry searchable fragment HTML and a stable fragment source, so that downstream code does not need to reinterpret parser nodes or source-specific values.
13. As a package maintainer, I want candidate identity, duplicate suppression, card adapters, and parser nodes to stay private, so that the public API remains deep and maintainable.
14. As an Algolia customer on a strict public plan, I want every complete final record to remain at or below 9,999 compact UTF-8 bytes, so that indexing does not depend on plan-specific or ambiguous limits.
15. As a Ghost publisher, I want oversized legacy groups split only between whole extraction fragments, so that indexed HTML is never arbitrarily sliced or truncated.
16. As a Ghost publisher, I want continuation records to have deterministic stable IDs, so that repeated indexing produces predictable objects.
17. As an operator, I want an indivisible oversized record to fail locally before any Algolia write, so that a batch does not leave partial new state.
18. As an operator, I want actionable size errors containing content and source context, so that I can identify the record and policy constraint that failed.
19. As an operator, I want an update to replace all earlier records for the affected Ghost content, so that stale continuation records cannot survive when content shrinks.
20. As a Ghost publisher, I want a safe default excerpt and selected Ghost metadata repeated in every record, so that result UIs can render useful context without reconstructing it.
21. As a Ghost publisher, I want to choose from an allowlist of optional Ghost fields and aliases, so that I can tailor result records without arbitrary code or paths.
22. As a Ghost publisher, I want aliases and configured fields validated before network access, so that configuration mistakes cannot partially fetch or write data.
23. As a Ghost publisher, I want correctly typed false, 0, and empty-string values preserved, so that valid Ghost values are not mistaken for missing data.
24. As a Ghost publisher, I want missing optional relations and scalars represented deterministically, so that every record has a predictable projection shape.
25. As a Ghost publisher, I want content with no extraction fragments to produce one valid fallback record, so that a published item remains discoverable by title and projected metadata.
26. As a package maintainer, I want protected identity, navigation, fragment, and ranking fields to be immune to projection configuration, so that essential record invariants cannot be overwritten.
27. As an Algolia administrator, I want relevance, search, retrieval, facet, snippet, and ranking settings to remain user-owned, so that selecting record fields does not silently reconfigure my index.
28. As a Netlify operator, I want publish and unpublish handlers to synchronize records without reading or writing settings, so that incremental webhooks cannot reset deliberate index configuration.
29. As a Netlify operator, I want publish handlers to require only record-mutation permissions after one-time index setup, so that runtime API keys can use narrower ACLs.
30. As a CLI user, I want omitted Ghost content type selection to retain posts-only behavior, so that existing configuration keeps its current meaning.
31. As a CLI user, I want to select posts, pages, or both through one validated configuration field, so that I can index the Ghost content types my site uses.
32. As a CLI user, I want posts and pages paginated independently to completion, so that one content type cannot truncate the other.
33. As a CLI user, I want --limit and --page to remain partial upserts for each selected type, so that targeted runs never delete unseen records.
34. As an operator, I want a complete automatic run to reconcile only package-managed content, so that stale managed records are removed and unrelated index records remain untouched.
35. As a Ghost publisher, I want posts and pages with the same slug to coexist in one index, so that valid Ghost content is not overwritten or deleted by slug collision.
36. As an operator, I want Ghost content identity based on the stable Ghost ID, so that grouping, replacement, deletion, and distinct results do not depend on mutable slugs.
37. As a Ghost publisher, I want published, edited, unpublished, and deleted posts and pages synchronized by explicit type-specific webhook events, so that incremental state matches Ghost publication state.
38. As a current Netlify user, I want existing post handler URLs and valid envelopes to remain compatible, so that enabling page support does not break established webhooks.
39. As a Netlify user, I want parallel page handler exports with documented event mappings, so that page synchronization can be configured without generic ambiguous handlers.
40. As an operator, I want webhook transformations fully validated and size-checked before Algolia is mutated, so that an invalid event cannot partially replace content.
41. As an operator, I want a preflighted one-time migration from legacy slug-managed records to Ghost content identity, so that page support can be enabled without guessing which records are safe to change.
42. As an operator, I want ambiguous legacy state to stop the migration before writes, so that unrelated or unclassifiable records are never deleted.
43. As an operator, I want an index copy and explicit maintenance/deployment order before identity migration, so that I can restore the prior package and index state if verification fails.
44. As a maintainer, I want deterministic pull-request tests to use reviewed Ghost-rendered fixtures rather than mutable live content, so that CI remains repeatable, offline, and privacy-bounded.
45. As a maintainer, I want a non-blocking live content smoke test to detect structural drift, so that changing Ghost output is observed without destabilizing pull requests.
46. As a Ghost content owner, I want the smoke workflow to log only aggregate structural signatures and counts, so that editorial content, identifiers, URLs, HTML, and credentials never enter logs or artifacts.
47. As a maintainer, I want new live signatures to trigger human investigation rather than automatic fixture changes, so that mutable external content cannot rewrite test evidence.
48. As a maintainer, I want every behavior change to pass a scoped clean-checkout dry run before publication and public-artifact verification after publication, so that Nx dependency side effects and package contents are known before consumers adopt it.
49. As a maintainer, I want dependency-only releases identified separately from behavior-changing releases, so that users can understand why a package version changed.
50. As a maintainer, I want source product requests closed only after the relevant npm versions and public seams are verified, so that planning or merged code is not mistaken for delivered behavior.
51. As a maintainer, I want each stage to have an explicit rollback seam, so that a failed package, workflow, adapter, or index migration can be recovered without unpublishing a version.
52. As a maintainer, I want all new source, tests, build tooling, and smoke tooling authored in strict TypeScript, so that the revived system has one enforceable source-language baseline.

Implementation Decisions

• Treat finalized Wayfinder issue Plan a maintained HTML-to-Algolia extraction pipeline #186 and its closed decision tickets as the approved product and compatibility contract. This PRD is the delivery handoff; it does not imply that any planned behavior has shipped.
• Preserve the architectural boundary: the HTML extractor accepts rendered HTML and returns ordered extraction fragments; the fragmenter owns grouping, Ghost content projection, size policy, deep links, final IDs, and Algolia record construction; the indexer owns Algolia operations and settings; the CLI and Netlify packages are adapters.
• Author all new source, tests, build tooling, prototypes retained for implementation, and live-smoke tooling in strict TypeScript. Published JavaScript, source maps, and declarations are generated artifacts and are never hand-edited.
• Build @tryghost/algolia-html-extractor as a new public package using a clean-room state machine on pinned parse5@7.3.0. Keep the parser behind the package boundary. Prevent an automatic parse5 8 upgrade until the synchronous CommonJS contract is intentionally revisited.
• The compatibility extractor exposes one named synchronous extract(renderedHtml) operation at the package root, with no default export, class, factory, options, parser injection, or public subpath. It ships equivalent generated ESM and synchronous CommonJS artifacts and declarations.
• Compatibility extraction emits immutable fragments in document order with searchable HTML, source text, heading context, anchor, emitted position, heading rank, and the minimal source tag needed for legacy preformatted merging. Non-string runtime input throws native TypeError; HTML5 parsing and recovery handle empty or malformed strings.
• Freeze compatibility at exact final Algolia record parity. This includes legacy selected elements, document order, heading reset and weight behavior, direct and descendant anchors, carried-forward anchors, positions, strict anchor grouping, merging of null or repeated anchors, text-only preformatted merging, deep links, and final group-based IDs. Abandoned-package internals such as DOM nodes, selector APIs, MD5 IDs, constructors, and unused dependencies are not contractual.
• Establish a compact immutable Ghost-rendered evidence corpus with separate legacy-selected-flow, intentional-semantic-gap, and observed Ghost-card-boundary fixture families. Generate it from controlled synthetic Ghost source with pinned renderer provenance; never copy live editorial content into fixtures.
• Add richer searchable rendered meaning only in a separately released extractor version. Preserve the compatibility stream and add the smallest qualifying semantic units for table headers, captions, citations, blockquote fallback content, meaningful image alternatives, and fixture-backed authored Ghost-card roles. Compatibility descendants take precedence over duplicate rich containers. A supported heading-only card gets one fallback fragment; a card-only item with no qualifying meaning gets no placeholder. Exclude controls, runtime state, provider chrome, arbitrary URLs, decorative content, unknown-wrapper text, and explicitly hidden or non-semantic subtrees.
• The richer extractor keeps the same named synchronous operation but replaces the compatibility-only source tag with a discriminated fragment source. Element sources describe their tag and whether they are ordinary content or a card-heading fallback; attribute sources describe meaningful image alternative text. Every emitted fragment carries searchable fragment HTML.
• Keep discovery candidates, parser-node identity, semantic-container identity, normalization keys, precedence, suppression reasons, and card adapters private. Compare rich candidates only inside their local semantic container after Unicode NFC normalization, leading/trailing trimming, collapse of each internal JavaScript-whitespace run to one ASCII space, and locale-independent String.prototype.toLowerCase(). Retain punctuation and diacritics, preserve returned source values, and never suppress compatibility fragments or deduplicate across separate local containers.
• Introduce the deep fragmenter operation createAlgoliaRecords(ghostContent, options?). Keep transformToAlgoliaObject and fragmentTransformer as deprecated compatibility wrappers with their exact current behavior until an explicitly versioned future decision removes them.
• Support optional ignoreSlugs and one JSON-serializable Ghost content projection shape through the deep fragmenter operation. contentProjection contains a required fields array whose entries are either an allowed source name or a {source, as} alias pair, plus an optional customRanking array whose entries are {source, as} pairs; there are no patch lists or schema-version field. The CLI uses a top-level contentProjection; canonical Netlify handlers read ALGOLIA_CONTENT_PROJECTION JSON. Validate configuration before Ghost fetching or Algolia writes.
• Protect record fields that carry physical identity, Ghost navigation, grouped fragment content, heading context, anchors, and required ranking values. Projection cannot omit, alias, or overwrite protected fields. When Ghost content identity ships, contentId and contentType also become protected and repeated in every derived record.
• With no explicit projection, repeat image, tags, authors, and Ghost's computed excerpt in every record. excerpt is additive display text and never replaces fragment HTML; custom_excerpt is not enabled by default.
• When a projection is supplied, its required fields list is the complete desired optional set and may be empty. The exact initial allowlist is image, tags, authors, excerpt, custom_excerpt, feature_image_alt, feature_image_caption, canonical_url, featured, visibility, created_at, updated_at, published_at, and reading_time. Preserve the existing image mapping and reduced tag/author relation shapes. Exclude raw formats, plaintext bodies, code injection, frontmatter, internal IDs, SEO/social duplicates, richer relation objects, and arbitrary pass-through fields.
• Allow one flat alias per optional field matching ^[A-Za-z][A-Za-z0-9_]*$. Reject unknown or repeated sources, repeated outputs, collisions with protected fields, the customRanking container, canonical allowlist names, or Algolia-reserved names, and reject dots, leading underscores, wildcards, expressions, arbitrary paths, and executable mappers.
• Always emit fragment heading and position ranking. Allow configured sibling ranking inputs only from featured and reading_time, each with a validated output alias; never replace heading/position, coerce timestamps, evaluate expressions, or accept callbacks.
• Normalize missing optional scalars to null and missing tag/author relations to empty arrays. Preserve valid falsey values. Copy correctly typed values without coercion or sanitization; reject present values of the wrong documented type. Consumers must escape display text, while fragment HTML remains HTML.
• Emit one deterministic fallback record when Ghost content has no extraction fragments. It carries the content identity/navigation/title/projection, empty fragment HTML and headings, no anchor, base URL, and the finalized fallback ranking.
• Apply deterministic record-size handling after legacy anchor grouping. Every compact complete record must be no larger than 9,999 UTF-8 bytes, measured across the entire JSON record including metadata, attribute names, escaping, and physical object ID.
• Greedily pack whole extraction fragments in group order. Never split inside serialized HTML, truncate meaning, drop required metadata, or vary limits by Algolia plan. The first chunk retains the legacy group ID; continuation records append stable numeric suffixes. Repeat projection, heading, anchor, URL, and rank context in every chunk, using the first contained fragment's source position for each chunk.
• Remove ignored slugs before full protected-field and size validation, while still requiring each ignored item to have a valid slug so the exclusion decision is deterministic.
• Preflight the complete output for the relevant content/batch before any Algolia mutation. On any deterministic failure, throw one FragmenterError before returning records or making an Algolia request; never return a partial array. The error exposes INVALID_POLICY, INVALID_GHOST_CONTENT, or RECORD_TOO_LARGE, all deterministic issues in input order, and actionable size context. Leave unexpected defects as native errors.
• Replace all prior records for affected content before saving a validated new set, so shrinking or regrouped content cannot leave stale continuation records. The later Ghost content identity route changes this operation from slug to stable content ID.
• Do not let Ghost content projection implicitly alter Algolia settings. Searchable attributes, retrieval, facets, snippets, ordering, and ranking remain user-owned unless an explicit settings/migration step states otherwise.
• First delivery slice: change Netlify publish handling to initialize the index and mutate records without applying or reading settings. Remove settings ACL requirements from runtime setup and document one-time index configuration. Retain the current slug facet prerequisite until the identity migration changes it.
• Release the compatibility extractor as 0.1.0 before any public package depends on it. From a clean Node 24 checkout of main, run pnpm ship 0.1.0 --projects=@tryghost/algolia-html-extractor --first-release --dry-run, inspect the result, then run the identical command without --dry-run. Because npm trusted publishing requires the package to exist, use the finalized exact-package/exact-version upstream bootstrap in the reviewed publish workflow, immediately configure OIDC trusted publishing after the first release, revoke and remove the temporary token path, and never publish locally.
• Cut the fragmenter over to the compatibility extractor in a patch release with exact final-record parity, retained wrappers, clean packed-package module checks, and verified Netlify tracing. Select Netlify directly when its manifest or tracer changes; otherwise only Nx-created application bumps are dependency-only. Every dry run and real ship command must use the identical direct project set from a clean Node 24 checkout of main.
• Bootstrap the live smoke workflow in two reviewed changes: first a manual-only strict-TypeScript workflow and offline-tested smoke program with no baseline-writing path; after an upstream-main dispatch and independent aggregate-only review, record only reviewed signature IDs/counts and enable the daily schedule.
• Run the live smoke daily at 04:17 UTC and by manual dispatch only. Guard the authenticated job with exactly github.repository == 'TryGhost/algolia' && github.ref == 'refs/heads/main'; give GITHUB_TOKEN only contents: read; pin actions to complete commit SHAs; disable persisted checkout credentials; expose the repository-scoped Content API key only to the execution step; and keep the approved Ghost origin and API version explicit. Do not make the workflow a pull-request or branch-protection check and do not grant issues: write.
• Read published posts and pages independently for smoke evidence, regardless of the later product selection default. Paginate each stream completely. Normalize content in memory to approved structural facts and hashes of normalized structure only. Never log or upload credentials, request URLs, response bodies, HTML/text, editorial identifiers or values, link/media values, or hashes of editorial content.
• Fail the standalone smoke workflow for transport, schema, privacy, determinism, extractor-invariant, or unseen-signature errors. Report missing signatures and count changes without failing solely because mutable editorial content changed. Upload no live artifact, persist no live-result cache, and never auto-update baselines/fixtures or open issues from the workflow.
• TryGhost/algolia maintainers own first triage. The maintainer who creates or most recently changes the cron must enable and own failed-workflow notifications, inspect the sanitized summary, and rerun on main before escalation. Credential, operational, schema/pagination, structural, and extractor failures are then routed to their finalized human owners; only a human may create one focused issue after classification.
• Release the deep fragmenter record builder, projection policy, atomic validation, stable errors, fallback record, and deterministic 9,999-byte packing in a fragmenter minor before any adapter activates it. Keep consumers on exact wrappers during this stage.
• Adopt the deep record contract in the CLI next. Validate projection before any network access, retain posts-only enumeration at this stage, send no Content API fields parameter, request include=tags and/or include=authors only when those optional relations are enabled, preflight the complete batch, and replace prior records for affected legacy slugs before saving. Release as a CLI minor.
• Adopt the same record contract in the existing post webhooks after the CLI. Parse and validate Netlify projection configuration before writes, transform and preflight a complete replacement, replace prior post records, and continue to make no settings calls. Release as a Netlify minor.
• Release richer extraction as one coordinated dependency cut: a minor extractor version with the discriminated source and a minor fragmenter version that consumes richer fragments through the deep API while deprecated wrappers filter to the exact compatibility stream. Direct extractor consumers must treat this as a breaking pre-1.0 minor and coordinated app dependency releases must be verified together.
• Add page-capable library primitives without activating adapters: the fragmenter accepts explicit Ghost content type and stable Ghost content ID, while the indexer gains content-identity browse, replace, delete, and reconciliation operations. Preserve existing slug APIs and posts-only defaults during this dormant minor release.
• Add top-level contentTypes selection to the CLI with allowed values post and page. It must be nonempty and duplicate-free; omission remains posts-only. Do not add a CLI override. A Page means a published Ghost Page resource, not custom or dynamic routes.
• Fetch posts first and pages second through independent pagination streams. In a complete automatic run, request 100 items per page, follow each stream's pagination to completion, and retain the existing delay. --limit continues to accept only integers from 1 to 100; --page continues to require --limit; together they select one page independently for every selected Ghost content type. Any run using either flag remains a partial upsert that cannot delete unseen records.
• Treat a complete automatic run as authoritative only for package-managed Ghost content. After full validation and preflight, reconcile stale managed records and newly disabled types, save selected records, and leave unrelated index objects untouched. Existing slug skips remain slug-wide across selected content types.
• Use stable Ghost content ID as logical grouping, replacement, deletion, and distinct identity. Keep the physical fragment object ID scheme for Algolia upserts and keep slug as navigation metadata. Add protected contentId and contentType fields derived from the known endpoint or type-specific webhook adapter, never inferred from an incomplete object.
• At the identity migration, replace the required slug operational facet with filter-only facets for content ID and content type, and change attributeForDistinct to content ID. Keep other relevance settings user-owned. The complete enlarged record remains within the 9,999-byte ceiling.
• Before any in-place complete identity migration, classify existing records as valid new managed records, exact legacy managed records, unrelated records, or ambiguous records. Fail before writes on ambiguity. A partial run against an unmigrated index must fail with complete-migration instructions.
• Require the migration operating order: pause Netlify writes, copy the index for rollback, preflight all new records, run the complete CLI migration, verify managed identity/type/census/stale-record/unrelated-record invariants, then deploy and enable identity-aware handlers. Apply the required settings after successful record reconciliation so a failed record phase leaves old settings usable. If classification cannot make in-place migration safe, build and validate a fresh index and switch configuration.
• Add explicit type-specific incremental synchronization only in @tryghost/algolia-netlify. Preserve the existing post-published and post-unpublished basenames, exports, envelopes, and valid behavior, and add parallel page-published and page-unpublished exports.
• Map published and published-edited events to the type's publish handler using the current resource; map unpublished and deleted events to the type's unpublish handler using Ghost content ID from current and then previous as fallback. Do not register generic edited events because they include unpublished edits. Do not re-fetch webhook resources through the Content API.
• Publish/edit handlers validate, transform, size-check, and preflight the full current resource before replacing all prior records for its content ID. Unpublish/delete handlers delete all records for that content ID. No webhook handler reads or writes settings.
• Use independent Nx releases and always run from a clean Node 24 checkout of main, inspect a scoped dry run, and use the identical project/version arguments for the real ship command. Treat observable-output-preserving replacements, focused defects, and dependent-only updates as patches; treat new/changed public APIs, record/configuration shapes, richer semantics, settings, or webhook exports as minors while packages remain pre-1.0. Re-read live package tags before every ship; illustrative versions in the Wayfinder release plan are not reservations.
• Every behavior-changing implementation PR and release handoff includes release notes covering user-visible behavior, defaults, configuration or environment changes, affected records/settings, compatible versions, migration order, source issues, and rollback. Dependency-only releases explicitly say they contain no direct behavior change.
• Verify every public release from a clean Node 24 checkout using the repository's pinned package manager, complete test/typecheck/lint/coverage gates, relevant packed-package acceptance, signed release commit and tags, Publish workflow success, and clean-install/npm inspection of version, dependencies, exports, provenance, and tarball contents.
• Never roll back by reusing or unpublishing a version. Roll forward with a new patch, switch consumers to the previous compatible package set, disable/revert a workflow, or restore/switch to the retained index copy as appropriate.
• Close the original product requests only after the relevant registry versions and packed/public behavior are verified. A planning resolution, implementation commit, or merged pull request alone is not delivery.

Testing Decisions

• Good tests assert observable behavior at the highest stable public seam and replace only true external systems. They do not assert private parser nodes, candidate IDs, internal helper calls, file layout, or transient implementation structure. Required verification is deterministic and offline.
• Test the compatibility extractor through its public named operation and through exact final-record differential fixtures. Cover document order, selection, headings, anchors including carry-forward behavior, serialization/entities/malformed HTML, foreign and void elements, nested selected nodes, positions, ranks, and empty/malformed input.
• Test the packed extractor package from clean temporary consumers in both ESM and CommonJS, including runtime exports, synchronous invocation, declaration resolution, source maps/tarball contents, and the absence of parser types from the public seam. This extends the repository's packed-package acceptance style.
• Test the immutable Ghost-rendered corpus with provenance and integrity checks, literal extraction-fragment expectations, complete final-record expectations, compact UTF-8 byte counts, and privacy-boundary validation. Fixture review must remain independent from generation.
• Test richer extraction through public emitted fragments, not private candidates. Cover smallest qualifying semantic units, heading inheritance, image-alt escaping, card-heading fallback, local duplicate suppression, compatibility-fragment non-suppression, hidden/non-semantic exclusions, unknown-card behavior, and uniform searchable fragment HTML.
• Test createAlgoliaRecords as the deep fragmenter public seam. Cover default and explicit projections, the exact allowlist and alias regex, protected/canonical/reserved collision rejection, correct falsey values, missing values, wrong types, relation reduction, featured/reading_time ranking siblings, ignored-slug ordering, fallback records, repeated projection fields, thrown stable ordered FragmenterError issues, and no partial return.
• Retain exact tests for both deprecated fragmenter wrappers to prove that adding the deep API, richer extractor, or page-capable primitives never changes their existing output.
• Test record size using Buffer.byteLength(JSON.stringify(record), 'utf8') across the complete record. Include ASCII and multi-byte text, JSON escaping, metadata-heavy records, headingless content, exact near-boundary records, splittable groups, indivisible fragments, oversized repeated metadata, deterministic continuation IDs, per-chunk source positions, and no partial output on failure.
• Test the CLI at the process boundary against the existing local Ghost replay server while replacing only the external Algolia requester. Assert configuration validation occurs before any network access, request pagination/order, field includes, complete request/record/settings sequences, partial-run behavior, and stable process outcomes. This follows the current Ghost 6 CLI acceptance seam.
• For projection adoption, test that the CLI sends no Content API fields parameter, enables tag/author includes only when projected, preflights the whole batch, replaces earlier managed records before save, never leaves continuations after shrink, and does not change settings because of projection.
• For page selection, test independent post/page pagination, posts-first order, omission as posts-only, every valid selection, invalid/duplicate/empty selections before network access, per-type --limit/--page, slug-wide skips, same-slug post/page coexistence, complete-run reconciliation, partial upsert-only behavior, newly disabled types, and preservation of unrelated records.
• Test legacy identity migration at the CLI public seam with valid-new, exact-legacy, unrelated, and ambiguous record sets. Prove ambiguity and partial-unmigrated runs stop before writes; record reconciliation precedes settings; verification detects stale continuations or census mismatches; unrelated objects remain byte-equivalent; and a copied/fresh-index route can be selected without weakening classification.
• Test Netlify through the native Request/Response handlers and real fragmenter/indexer code while replacing only the external Algolia requester. Retain current authentication, activation, user-agent, envelope, controlled-error, and basename behavior as prior art.
• For the settings fix, prove publish invokes no settings request and works with a mutation-only key contract. Prove the required one-time filterOnly(slug) setup through the existing CLI/indexer public settings seams; introduce no Netlify setup endpoint or operation.
• For projection and size handling, prove handler configuration is parsed and all deterministic policy/content/size validation completes before any Algolia request. Separately assert delete-before-save ordering, complete replacement for shrinking content, surfaced external failures without a rollback-atomicity promise, and no settings call.
• For content-type synchronization, exercise the four exact public handlers at the native request seam. Cover current/previous ID fallback, post compatibility, page envelopes, same-slug independence, replacement/deletion by content ID, malformed bodies, and no Content API re-fetch. Separately verify the documentation/configuration contract maps the eight approved Ghost events to those handler basenames and never registers generic edited events.
• Test the indexer through its public factory/operations with the requester replacement. Cover initialization versus explicit settings, content-identity browse/replace/delete/reconciliation, escaping/filter construction, pagination, unrelated-record preservation, required identity settings, external error mapping, and no hidden network bypass. This extends current indexer contract tests.
• Test the live smoke program offline with synthetic Content API pages and an injected or local transport seam. Cover posts/pages pagination, repeated or malformed pagination, origin/version/redirect rejection, deterministic normalization, allowed signature fields, known/unseen/missing/count-change outcomes, aggregate-only summaries, no artifact/cache/baseline-write path, and redaction of keys, request/response material, raw errors, HTML/text, editorial values, URLs, and content-derived hashes. Statically verify the exact upstream-main guard, pinned actions, read-only permissions, disabled persisted credentials, execution-step-only secret scope, and absence of issues: write.
• Validate the workflow itself with manual upstream-main bootstrap: first merge manual-only code, dispatch only from main, have a second maintainer inspect the aggregate-only summary, then commit the reviewed baseline and enable the daily schedule. The live run is operational verification, not a deterministic PR assertion.
• At every release boundary, run the complete repository test suite, strict typecheck, lint/format checks, coverage gate, and relevant packed ESM/CommonJS/declaration/Netlify acceptance. Inspect the scoped Nx dry run and verify every direct and dependent project it selects.
• After publication, verify each npm artifact through registry metadata and a clean install. Confirm exports, declarations, dependencies, provenance, tarball boundaries, packed handler behavior, and the expected compatibility or behavior change before a consumer is migrated or a source issue is closed.

Out of Scope

• Embedding, invoking, or porting the abandoned Ruby extractor, or porting abandoned JavaScript implementation internals.
• A broad module-format migration or adoption of parse5 8; that requires a future intentional decision about synchronous CommonJS support.
• Changing the compatibility-first final record output to fix semantic gaps, size behavior, projection, page identity, or other known defects; those changes remain in their separately sequenced releases.
• Caller-configurable extractor selectors, parsing profiles, parser injection, generic card policy, exposed parser nodes, or public candidate/suppression diagnostics.
• Arbitrary Ghost field pass-through, executable mappers, nested/dotted aliases, raw content formats, plaintext bodies, code injection, internal identifiers, or automatic sanitization/coercion.
• Automatically changing searchable attributes, retrieval, facet, snippet, relevance, or ranking settings when projection fields are selected.
• Custom routes, dynamic routes, drafts, or other resources that are not published Ghost Post or Page Content API resources.
• A CLI command-line override for Ghost content type selection, type-specific slug exclusions, or generic webhook edited events.
• Re-fetching published webhook resources from the Content API or adding HTTP/Ghost/Algolia behavior to the HTML extractor.
• Expanding webhook authentication beyond preserving the existing public contract; the known optional-key security concern requires a separate focused decision and delivery effort.
• Making mutable main.ghost.is content, a live Content API read, or a live Algolia write a required pull-request check.
• Persisting live editorial content, URLs, identifiers, raw HTML/text, media targets, response bodies, or content-derived hashes in fixtures, logs, summaries, or artifacts.
• Automatically updating smoke baselines/fixtures or automatically creating tracker issues from drift.
• Publishing any package by hand, using unscoped ship commands for this effort, reusing/unpublishing a failed version, or treating illustrative versions as reserved.
• Closing source product requests before their registry versions and public behavior are verified.
• Unrelated Renovate work or extraction changes prompted solely by future smoke drift; new evidence starts a focused follow-up decision or implementation effort.

Further Notes

• This PRD is synthesized from the completed Wayfinder map in Plan a maintained HTML-to-Algolia extraction pipeline #186 and the approved resolutions it indexes. Plan a maintained HTML-to-Algolia extraction pipeline #186 remains the canonical decision ledger; this issue is the implementation handoff.
• The finalized route begins with the independent Netlify settings fix, then creates and releases the compatibility extractor before cutting over the fragmenter. Deep record policy, adapter adoption, richer semantics, dormant page primitives, CLI migration, and Netlify page synchronization follow in that dependency order.
• The route intentionally contains multiple independently reviewable and releasable slices. ready-for-agent means the PRD is fully specified; it does not authorize an agent to collapse the release gates, migrate consumers before their dependencies are public, or start a blocked later slice.
• The first unblocked delivery frontier is to stop Netlify publish events from rewriting Algolia index settings. Agents should claim and complete one frontier slice at a time, rechecking live package tags, tracker state, and published artifacts before continuing.
• The repository's canonical language distinguishes Ghost content, Ghost content type, Ghost content identity, Ghost content projection, extraction fragments, searchable fragment HTML, fragment source, HTML extractor, fragmenter, Algolia record, Ghost-rendered fixture, and live content smoke test. Implementations and release notes should keep those distinctions.
• Source requests covered by the route are Searchable attributes in the config file are not added to algolia #23, Extend algoliaPost with additional ghost post field #43, html tags in search results #148, and Cannot index pages instead of posts #4. Their closure gates are delivery-specific and remain later than planning completion.