feat(pipeline)!: add beforeDatasetWrite hook; split namespace normalization

[ddeboer]

Reworks the schema.org namespace normalization from #568 so the VoID analysis core no longer knows about namespace aliases – normalization is one opt-in plugin at the pipeline edge again, and the namespace handling is split by generality.

Why

#568 fixed the duplicate void:classPartition (netwerk-digitaal-erfgoed/dataset-knowledge-graph#334) but threaded namespace-alias awareness through the core: a per-stage merge transform, a class selector that canonicalized and co-located variants per batch, query markers, and self-describing void-ext chains. That coupled a consumer-specific concern into the analysis queries and stage machinery.

Two facts reframe it: merging per-class partitions needs a class’s http/https variants seen together, which beforeStageWrite (per stage/batch) can’t provide – hence the co-location leaking into the core; and datasets typically use a single schema.org namespace, so the cross-variant merge is for the rare mixed dataset, not the common case.

What

@lde/pipeline – a generic per-dataset hook, plus a generic namespace rewrite:

• New PipelinePlugin.beforeDatasetWrite: a QuadTransform run once over a whole dataset’s cross-stage output before write – the dataset-scoped sibling of beforeStageWrite. Implemented as a RunWriter decorator that streams every stage’s output through one AsyncQueue and honors the flush/reset/commit/abort run lifecycle; nothing is materialized. Domain-agnostic – dedup, roll-ups, or partition merging can use it.
• Keeps namespaceNormalizationPlugin / schemaOrgNormalizationPlugin, now generalized into a blanket, vocabulary-agnostic beforeStageWrite rewrite of a namespace’s IRIs in every term position (previously only void:class/void:property objects). This serves consumers that just standardize a namespace across a dataset’s own quads – e.g. mapping instance data to an application profile – with no VoID involved.

@lde/pipeline-void – the VoID partition merge, core reverted:

• New schemaOrgPartitionMergePlugin() (and generic namespacePartitionMergePlugin(aliases)): a beforeDatasetWrite plugin that canonicalizes void:class/void:property and re-mints/merges the duplicate partition nodes from their canonical components. A whole-dataset view resolves each partition’s cp → pp → dp chain from the other stages, so the queries need no markers or self-describe chains. Renamed so a VoID consumer can’t pick the plain rewrite and leave the duplicate nodes unmerged.
• The six analysis queries revert to plain SPARQL; the namespaceAliases stage option, class-selector canonicalization/expansion, query markers and void-ext self-describe chains are gone.
• The merge streams: only partition quads are buffered (bounded by the summary), everything else passes through.

Why the split

The generic rewrite and the merge share only the canonicalizeIri primitive. Rewriting void:class objects alone would still leave two void:classPartition nodes for one class, so the merge (VoID-specific: it knows the partition vocabulary and the shared partition-IRI minting) can’t be replaced by the rewrite. And the merge’s canonicalization is deliberately selective – it leaves void:vocabulary and entity-properties in the source namespace – whereas the generic rewrite is a blanket pass. So the two stay separate plugins, in the package each belongs to.

Correctness

Single-namespace datasets (the common case) keep every count exact – the plugin only renames and re-keys. A dataset that genuinely mixes both namespaces on one property has its void:distinctObjects summed (an over-count for shared objects), which is documented and not optimized for; entities/triples stay exact under subject-disjointness.

Tested: the beforeDatasetWrite hook (streaming, flush, reset, abort) in @lde/pipeline; the generic rewrite plugin (all term positions) in @lde/pipeline; the merge transform + plugin (incl. the MD5 minting matching the queries, single-namespace exactness, and the mixed over-count) in @lde/pipeline-void; and an endpoint-backed test running the real plain queries through the plugin end-to-end.

See ADR 7. Follow-up: dataset-knowledge-graph switches to plugins: [schemaOrgPartitionMergePlugin(), provenancePlugin()].

BREAKING CHANGE: @lde/pipeline’s namespaceNormalizationPlugin / schemaOrgNormalizationPlugin now rewrite every matching IRI in any term position via beforeStageWrite, not just void:class/void:property objects. The VoID partition merge is now schemaOrgPartitionMergePlugin / namespacePartitionMergePlugin in @lde/pipeline-void, and voidStages no longer accepts namespaceAliases – add the plugin to the pipeline plugins instead.

https://claude.ai/code/session_01H8MVbsXSoNbYREYMFLy6Eg