JSON dispatch: application/json falls back to HtmlExtractor, silently ignoring jsonpath specs

[gregoryfoster]

Rewritten 2026-08-19 in a backlog staleness pass. The problem is real and confirmed in current code, but it is more severe than "an enhancement" and the implementation instructions pointed at a package that no longer exists plus a policy that has since been repudiated. The extractor itself is now CannObserv/cannobserv#354; this issue keeps the Watcher half. Original body preserved at the bottom.

The problem (confirmed)

src/core/registry.py maps five essences; dispatch is total and everything else falls back to HTML:

_DEFAULT_EXTRACTOR_MAP: dict[str, type[Extractor]] = {
    "text/html": HtmlExtractor,
    "application/xhtml+xml": HtmlExtractor,
    "application/pdf": PdfExtractor,
    "text/csv": CsvExcelExtractor,
    "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet": CsvExcelExtractor,
}

application/json is not listed, so JSON runs through HtmlExtractor.

Severity correction: not benign

The original said "HTML fallback is non-crashing today, so this is an enhancement, not a regression." Non-crashing, yes. Harmless, no — verified against installed co-core 0.10.0:

raw = b'{"items": [{"title": "Rule 5", "status": "active"}]}'
HtmlExtractor().extract(raw, {"algorithm": "jsonpath", "selector": "$.items[*].title"})
# -> one chunk, text = the entire document. Selector ignored.

Identical to passing config=None. Archiver's schema enforces {json: jsonpath} as a content-kind family, so a JSON InfoSource legitimately carries a jsonpath spec — which then reaches an extractor that cannot honour it.

Result: a silent false-positive generator. An item whose spec says "watch $.items[*].title" fingerprints the whole document instead, so any unrelated field — a timestamp, a counter — reads as a change to the watched content. Chunks are non-empty, so #258's empty-extraction guard never trips and health stays OK. It fails quietly while appearing to work.

Whether this is live today depends on whether any watched item currently carries a jsonpath spec — worth checking before deciding priority, since it determines if this is latent or actively producing noise.

Corrections to the original instructions

The Ask said:

src/core/extractors/ — new JsonExtractor (mirror to /home/exedev/archiver/src/core/extractors/ per the mirrored-content-acquisition policy)

Both halves are wrong now:

1. There is no src/core/extractors/ in Watcher. Extraction moved to co-core (co_core.pure.extract.{html,pdf,csv_excel}) — confirmed: the package has no json module. Writing the extractor is a cannobserv change, filed as CannObserv/cannobserv#354.
2. The mirroring policy is repudiated. AGENTS.md: "Nothing in src/ mirrors to Archiver (Refactor brittle Watcher↔Archiver "mirror discipline" for shared content-acquisition code #159, Phase 1c: adopt co-core v0.5.0 content-acquisition; delete the src/core mirror #236) — don't reintroduce a sync obligation." Following that instruction would resurrect a discipline that was deliberately removed. Do not.

Watcher's half (this issue) — blocked on cannobserv#354

Small once upstream lands:

• Import JsonExtractor and map application/json → it in _DEFAULT_EXTRACTOR_MAP (src/core/registry.py).
• application/*+json (vendor JSON) essence handling — this is genuinely Watcher's, since essence resolution lives in src/core/media_type.py. Decide whether application/ld+json, application/vnd.api+json etc. normalize to application/json for dispatch, or need explicit entries. A structured-suffix rule (+json ⇒ json family) is the obvious approach and mirrors the existing .xlsx/.pdf extension-tiebreaker pattern.
• Routing tests: application/json → JsonExtractor; a +json vendor type → JsonExtractor; unmapped essence still → HtmlExtractor.
• Check whether content_media_type auto-detection (Auto-detect content type from URI response; evaluate MIME vs enum #168) resolves JSON correctly, or whether the tiebreaker needs a .json extension entry alongside .pdf / .xlsx.

Acceptance

• application/json dispatches to a real JSON extractor, not HTML
• application/*+json handled deliberately (mapped or explicitly out of scope)
• Routing tests cover mapped, vendor-suffix, and fallback cases
• A jsonpath spec actually narrows the fingerprint to the selected nodes

Related

• CannObserv/cannobserv#354 — the extractor itself; blocks this
• Auto-detect content type from URI response; evaluate MIME vs enum #168 — the dispatch mechanism this extends; also deferred magic-byte/extension hardening (co-core's side) and drift detection
• All-empty extraction is recorded as a content change with OK health #258 — why an unmatched selector should yield empty chunks rather than a phantom change

---

Original body (2026-06-23)

Background

#168 (slice 2) made content_media_type drive extractor dispatch via ServiceRegistry.get_extractor (src/core/registry.py). Dispatch is total: any essence not explicitly mapped falls back to the HTML extractor.

application/json has no dedicated extractor, so JSON targets are currently run through HtmlExtractor. Archiver's declared content-kind family already includes json (jsonpath → json), but Watcher has no JSON extractor to match.

Ask

Add a JSON extractor (e.g. canonicalize/pretty-print + structural chunking, or JSONPath-aware extraction aligned with Archiver's jsonpath algorithm) and register it:

• src/core/extractors/ — new JsonExtractor (mirror to /home/exedev/archiver/src/core/extractors/ per the mirrored-content-acquisition policy).
• src/core/registry.py — map application/json (and likely application/*+json) → JsonExtractor.
• Tests: routing (application/json → JsonExtractor) + extraction behavior.

Notes

• HTML fallback is non-crashing today, so this is an enhancement, not a regression.
• Consider application/*+json (vendor JSON) essence handling at the same time.
• Related: magic-byte/extension hardening and drift detection were also deferred from Auto-detect content type from URI response; evaluate MIME vs enum #168.