Restore change detail (unified diff + chunk semantics) once Replicator provides a persistent content store

[gregoryfoster]

Reviewed 2026-08-19 in a backlog staleness pass. Still blocked — but the blocker is not quite what this issue says, and #221 (its other dependency) has closed. Two corrections and a design direction added below; everything else stands and remains accurate.

Correction 1 — Replicator does not solve this as assumed

The header says the content store is "forthcoming in the TBD Replicator service". Replicator now exists and permanent replication shipped (replicator#29, closed — GCS per RepSpec, via AsyncGcsDriver.create_if_absent). That does not deliver what this issue needs:

• It stores raw fetched bytes, not the extracted text the fingerprint is computed over. Diffing raw HTML is not the same product as diffing extracted content, and this issue's "Diff target" question below is precisely about that choice.
• It is per-RepSpec, so coverage is partial. This issue's own acceptance criterion — "Diff availability does not depend on whether the item is Archiver-linked" — fails against a per-RepSpec store the same way it fails against the Archiver-fetch option, and for the same reason.

So "wait for Replicator" is not by itself a plan. Something has to be specified.

Correction 2 — Watcher already holds the text it needs to store

src/workers/pipeline.py:

content_bytes = "\n".join(c.text for c in result.chunks).encode()
fingerprint = "sha256:" + hashlib.sha256(content_bytes).hexdigest()

The extracted text exists in memory at apply time — it is what the fingerprint is computed over — and is then discarded. ChangeRevision persists only content_fingerprint, content_size_bytes, schema_version, captured_at.

This means deferred design question 1, option B (denormalize previous extracted text onto ChangeRevision, size-capped and compressed) requires nothing from Replicator and could be built today. It is recorded here as available, deliberately not chosen — see the direction below.

Direction — a Replicator-side sidecar, not a Watcher-side column

The preferred shape is for Replicator to store the full extracted text as a sidecar artifact alongside the blob it already holds, addressed by URI, rather than Watcher freighting text through its own database or the bus.

This follows the pattern already in force. Nothing puts content on Redis today — SourceRevisionObservedEvent on content.revisions carries blob_uri, extracted_fingerprint, spec_fingerprint, sizes and media types, but never the bytes. The bus is claim-check by design; the data plane is the blob store. A sidecar keeps that property, where a denormalized ChangeRevision column moves content into Watcher's Postgres and a bus-carried payload would move it into Redis. Both are the wrong direction.

The wrinkle to resolve, stated plainly: Replicator does not extract. It fetches raw bytes; extraction is Watcher's, because Watcher holds source_specs. So a sidecar requires either

• Watcher writes the sidecar into Replicator's store after extraction — needs a write path into a store Watcher currently only reads from, and blob_uri is a host-local file:// by contract, or
• Replicator gains extraction — a substantial boundary move that would need source_specs on the fetch command, and contradicts the current split.

The first is much the smaller change and fits replicator#7 (Object-store blob backend (claim-check) — remove the shared-filesystem data-plane coupling), which is already about giving the blob store a real addressable backend rather than a VM-local path. That issue is the natural place for this to land, and this issue should not be unblocked until it does.

Retention is the other half: blobs today are deliberately transient (TTL sweeper, BlobUsage ceiling). A diff sidecar must outlive its blob by at least one check interval, which is a different retention class and needs stating in whatever contract change carries it.

Status of the original dependency

#221 is closed (2026-07-10) — the dead surface this issue restores has been stripped as planned. The 89d64eb restore anchor below remains valid.

---

Blocked by: a persistent content store — forthcoming in the TBD Replicator service.
Depends on: #221, which removes the dead surface this issue restores.

Context

Phase 5 (#156) dropped the Snapshot/Change tables and the diff engine. ChangeRevision
stores a content fingerprint and nothing else (change_revision.py),
so at dispatch time watcher has no previous content to diff against.

The notification UI kept offering Diff snippet, Full diff, and Significance score
anyway. notify.py:188 hard-codes unified_diff = "",
and real change_detected metadata never carries significance / added / modified /
removed, so all three toggles have been inert in production since #156. Only the preview
appeared to work, because it was fed canned fixtures.

#221 strips that surface: the toggles, the ContentOptions fields, and the now-unreachable
plumbing all come out. This issue restores the capability once there is somewhere to read
previous content from.

Watcher is, for now, a fingerprint-shift notifier: it reports that a page changed,
not what changed.

Restore anchor

All removed code is preserved in git at 89d64eb (the commit immediately preceding the
#221 strip). Restore from there rather than rewriting — in particular
_truncate_unified_diff_lines is hunk-boundary-aware and carries real test coverage that
would be tedious to reconstruct.

git show 89d64eb:src/core/notifications/content.py
git show 89d64eb:src/api/schemas/content_config.py
git show 89d64eb:src/core/notifications/preview_fixtures.py

Inventory of what #221 removes

Location	Removed
content_config.py	include_diff_snippet, diff_snippet_lines, include_diff_full, include_significance
content.py	_build_diff_text, _render_unified_diff_block, _normalize_unified_diff_lines, _truncate_unified_diff_lines, _format_chunks_changed, _compute_change_summary, _DEFAULT_DIFF_SNIPPET_CAP, the diff_snippet / diff_full / chunks_changed / change_summary context keys, and the unified_diff / diff_snippet_cap params on build_body + build_template_context
notify.py:185-201	the unified_diff = "" stub and its pass-through
preview_fixtures.py	compute_preview_unified_diff, _PREVIEW_PREVIOUS_TEXT, _PREVIEW_CURRENT_TEXT, and the added / modified / removed / significance fixture keys
default_templates.py	TemplateVariable entries for change_summary, added, modified, removed, diff_snippet, diff_full, chunks_changed, significance; and CHANGE_DETECTED_BODY_BLOCK_LINES
notification_form_content_body.html	the entire Changes toggle group
routes.py	the corresponding branches in _parse_content_config_from_form and the preview route

What restoring actually requires

Restoring is not re-adding the checkboxes. Three separate problems, in ascending cost:

1. Previous content must be readable at dispatch time. Today it exists only as an
   Archiver SourceRevision, plus a TTL-bounded scratch file (WATCHER_CACHE_DIR,
   default 600 s) that is written only when archiver_info_source_id is set —
   dashboard-created WatchedItems have none. This is what Replicator is expected to solve.
2. Unified-diff computation no longer exists. compute_unified_diff is gone from
   src/ entirely; only the rendering helper survived (and Notification Template UI/UX polish: event labels, Context layout, and strip the dead diff/significance surface #221 removes that too).
   Cheap to rebuild on difflib once (1) lands.
3. Chunk semantics are a rebuild, not a restore. added / modified / removed /
   significance / chunks_changed were produced by the diff engine Phase 5 deleted.
   Nothing in the current codebase computes them. Treat as a separate scope.

Deferred design questions

• Storage source. Read previous text from Replicator/Archiver at change time (a network
  hop on the change path), or denormalize previous extracted text onto ChangeRevision
  (size-capped, compressed)? The Archiver-fetch option silently yields no diff for
  WatchedItems without archiver_info_source_id — a worse failure mode than no diff at all,
  so whatever Replicator exposes must cover every item.
• Diff target. Extracted text (what the fingerprint is computed over) or raw bytes?
• Does significance come back at all, or is a unified diff sufficient signal?
• Default body. Notification Template UI/UX polish: event labels, Context layout, and strip the dead diff/significance surface #221 reduces the change_detected default body to the header block.
  Restoring a diff means re-deciding where it sits and whether it's on by default
  (verbose channels vs. Slack).

Acceptance

• A delivered change_detected email with Full diff enabled contains the diff.
• Diff availability does not depend on whether the item is Archiver-linked.
• The preview renders the same diff the dispatcher would send — the parity test added
  in Notification Template UI/UX polish: event labels, Context layout, and strip the dead diff/significance surface #221 still passes, now with diff fields on both sides.
• Restored toggles appear in the Content form only when they have an observable effect.

[gregoryfoster]

The sidecar direction is now posed as an explicit boundary question: CannObserv/replicator#69.

CannObserv/replicator#69

Rather than assuming Replicator will provide the store, that issue tests the proposal against Replicator's own boundaries charter (docs/contracts/replicator-boundaries.md) and reports honestly that the charter's current reading is probably "no":

• Test 3 (no domain vocabulary) passes cleanly — Replicator would hold an opaque, content-addressed artifact it never reads.
• Test 2 (cross-command coordination) does not fire.
• Test 1 (durable per-resource history) is where it fails. The charter requires Replicator's state be rebuildable by re-fetch, replay, or held in the broker. A previous-extracted-text sidecar is content-addressed but not rebuildable by re-fetch, because re-deriving it needs source_specs — and "Never Replicator's: … extraction specs" is a named line in that document.

The charter's own conditional-GET worked example is the precedent against it: ETags looked like fetcher state, test 1 sent them to the issuer, and Replicator gained the capability while holding zero bytes of per-URL history. The document explicitly says to cite that case when a proposal argues some small store would be simpler.

Why this matters here: this issue's stated blocker — "a persistent content store — forthcoming in the TBD Replicator service" — may never arrive, and not because of scheduling. If replicator#69 resolves no, this issue stops waiting and chooses between the two options that remain its own to make:

• A. Denormalize onto ChangeRevision (size-capped, compressed) — buildable today, needs nobody.
• C. Read from Archiver — already rejected above for the network hop and the archiver_info_source_id coverage gap, and worth re-examining only if archiver#118 changes that.

Either way the answer is worth having in writing. A no unblocks this issue's decision-making as effectively as a yes would unblock its implementation.

[gregoryfoster]

Unblocked — the boundary question this issue was waiting on is answered. CannObserv/replicator#69 closed 2026-08-20, recorded in Replicator's boundaries charter (docs/contracts/replicator-boundaries.md, Asked and answered).

Two rulings matter here:

1. No Replicator-held sidecar, ever. The "persistent content store — forthcoming in the TBD Replicator service" this issue's blocker assumed is not coming: a derived artifact Replicator cannot rebuild fails its charter twice over (not rebuildable without extraction specs, and an issuer-written artifact is a write surface into a store whose content-addressing depends on addresser and writer being the same process). Stop waiting on it.

2. What was granted instead — and it serves this issue fully: blob retention sized to check cadence (replicator#69 "option E"). REPLICATOR_BLOB_TTL_SECONDS runs from last reference and is already a published contract commitment (default 7 days) with a documented raise-on-consumer-request path. Unchanged content re-references its blob on every check, so it persists indefinitely; only a changed-away blob ages out, and the diff is needed at most one check interval after its last reference. So the requirement is just TTL ≥ slowest check interval + slack — if no cadence exceeds 7 days, Replicator needs zero changes; otherwise file a raise request on the Replicator repo.

The Watcher-side shape:

• Persist the previous blob_uri + blob_expires_at per item — two small columns, a reference rather than content (much lighter than the denormalize-onto-ChangeRevision option A, and none of its cascade/growth concerns).
• At change time, read the old blob (gs:// read support and the objectViewer grant already exist via blob_uri: support gs:// for replicator#7, and cap the unreadable-blob re-issue loop #275) and re-extract with current specs, then diff the spec-selected chunks. Re-extracting old bytes with current specs isolates genuine content change from spec change — where storing text extracted under old specs conflates them. Interacts cleanly with Conditional GET parts 2-4: store and replay validators, decide the re-fetch cadence #269's validator_source_key.
• Blob expired → no diff, same as today: the benign, documented fallback.
• Acceptance criterion satisfied: diff availability depends only on blob age, never on whether the item is Archiver-linked.

One horizon note: under Replicator's gcs backend, blob_expires_at is a floor, not an exact time (bucket lifecycle granularity) — safe direction for this use: the blob may outlive the horizon, never die before it.

The permanent-record ambition (canonical derived artifacts under a pinned Replication Specification) survived the discussion but left Replicator entirely — successor issue CannObserv/archiver#179 (contract-first content.process, leaning Observo as the processing infrastructure). This issue does not depend on it.

Cross-filed from the replicator#69 disposition.

[gregoryfoster]

Reviewed against the tree at 3414ce0. The 2026-08-20 direction is right — persist a reference and re-extract at change time — and it is buildable today. But its retention argument has two holes, both Watcher-side, both cheap to close, and both change what slice 1 has to contain.

What the current tree confirms:

• pipeline.py holds blob.blob_uri / blob.blob_expires_at at apply time and writes them only to the outbox row, which source_revisions_drain.py deletes on publish. Nothing durable retains the pointer.
• blobs.py already reads gs://, and GCS_BLOB_CREDENTIALS is present on the VM — the read path this issue needs exists.
• change_revision.py still stores four columns, none of them a pointer.

Finding A — "unchanged content re-references its blob on every check" is false for most real pages

The blob is content-addressed over raw bytes; the fingerprint is over extracted text. A page carrying a rotating nonce, timestamp, ad slot or CSRF token yields a new blob every check while the fingerprint sits still. The blob backing the previous change is then never re-referenced again, and dies REPLICATOR_BLOB_TTL_SECONDS after the check that produced it — not one check interval later. Diff availability degrades to "two consecutive changes fell within 7 days of each other", which for a quarterly-updating regulatory page means never.

Fix — touch-forward on the cache-hit branch. Where process_watched_item currently returns on an equal fingerprint, overwrite the last revision's blob_uri / blob_expires_at with this cycle's. Sound because an equal fingerprint means the newer blob re-extracts to the same text under current specs, so it is an equally valid diff base — and it was just referenced by a fetch, which is the clock Replicator actually runs (docs/STORAGE.md: TTL from last reference by a fetch, customTime stamped on the re-reference path). One UPDATE per check. This is what makes the "TTL ≥ slowest check interval + slack" requirement true rather than assumed.

Finding B — #269 conditional GET defeats the re-reference entirely

A 304 produces no bytes, so nothing touches the store. An item named in WATCHER_CONDITIONAL_GET_ENABLED that 304s for a week loses its diff base regardless of how often it is checked. Latent today — the variable is unset in production — but #269 and this issue are on a collision course, and the constraint should be written down before either ships wider.

Fix — validator keepalive. When blob_expires_at - now falls inside a margin, suppress the replayed validators for one cycle to force a 200. Cost: at most one unconditional fetch per TTL per item. It must go through the validator-clearing path in validators.py rather than around it — routing a suppression past that gate is exactly the wedge hazard #269 exists to prevent.

Recommended shape

Columns on ChangeRevision, not per-item — blob_uri, blob_expires_at, and spec_fingerprint. The first two are free at insert. The third is already computed in _extract_and_fingerprint and currently discarded with the outbox row; without it, re-extracting old bytes under current specs silently presents a spec-induced difference as a content change. With it, a diff across a spec boundary can be labelled or suppressed. Per-revision rather than per-item also gives history for free and needs no second writer.

Compute in the pipeline at change time, strictly non-fatal. After the revision insert, before dispatch_event_notifications. Every failure — BlobUnreadable, UnsupportedBlobScheme, ExtractionError on the old bytes — must degrade to no diff, notification still sent, health untouched. A raise there would abort the revision and mark ERROR, which is far worse than a missing diff. aread_blob, never read_blob — one process runs the API, the consumer and the tasks.

Carry the diff on event.metadata; nothing persists it. build_body renders per candidate and the text reaches notifier as an inline template; the audit row stores only results. So a large diff never lands in Postgres. Restore _truncate_unified_diff_lines and the render helpers from 89d64eb as planned — the cap belongs at render time, not at storage time.

Acceptance criterion 3 is already automated. tests/core/notifications/test_preview_fixtures.py::test_change_detected_matches_pipeline_metadata enforces fixture keys ⊆ emitter keys, and test_change_detected_has_no_diff_phantom_keys is its current negative. Adding unified_diff to both sides satisfies the parity criterion by construction rather than by a new test.

Scope — three slices, not one issue

1. Blob reference persistence. Three columns + migration, touch-forward, keepalive gate. No user-visible change; verifiable on its own by asserting the previous blob is readable at change time. This slice carries essentially all of the risk.
2. Diff restore. difflib recompute, restore the render helpers, toggles, ContentOptions fields and the Changes form group from 89d64eb, plus preview fixture parity.
3. Chunk semantics. New information since this issue was written: re-extracting the old blob yields the old chunks, so added / modified / removed / chunks_changed become genuinely derivable rather than a rebuild from nothing. Still a separate scope.

Deferred design questions — proposed resolutions

• Storage source: neither A nor C. Store a reference, not text, and no network hop to Archiver.
• Diff target: extracted text, re-extracted from the retained raw bytes under current specs — which is what makes the spec_fingerprint column load-bearing.
• Does significance come back: no. Close it as won't-do. Nothing has ever defined it, and a unified diff is the signal.
• Default body: decide with slice 2; the toggle-only-when-observable criterion already constrains it.

Preconditions and residuals

• All four live items resolve to ≤ 1d (one at 1h, the rest on the 1d system default) against a 7-day TTL, so no raise request on Replicator is needed. Worth a guard so a future item slower than TTL − slack is caught loudly rather than being silently diff-less.
• Backfill: the 34 existing revisions carry no blob_uri, so the first diff per item arrives on its second change after ship. Expected, not a defect.
• The title should change — Replicator is not providing the store, and never will (replicator#69). Something like Restore change detail by diffing against the retained blob.