Two proposals arising from reconciliation output: certainty on links[], and one authoritative list of identifier prefixes
Both of these come from the same use case, so I've filed them together — happy to split them if the working group would rather take them separately.
Context. World Historical Gazetteer's Map your Data workbench reconciles a user's table against gazetteers (GeoNames, Wikidata, Getty TGN, OpenStreetMap, OpenHistoricalMap, PeriodO, WHG's own contributed datasets) and exports the result as LPF, either for download or for contribution back to WHG. The output of a reconciliation is a set of closeMatch links — which the README describes as "the principal means of linking places and gazetteer datasets", and for which it points at WHG's reconciliation service. Two things about that output can't currently be expressed.
1. Allow certainty on links[]
Existing situation
certainty — "certain" / "less-certain" / "uncertain", following the Pleiades convention — is allowed on geometries and relations, and the v1.2.2 changelog (4 September 2022) records it being extended to when, "wherever it is used, as already allowed for the geometries and relations elements". #47, which proposed that, describes its earlier absence there as an oversight.
It is not allowed on links[].
Yet a closeMatch is exactly the kind of statement whose confidence varies, and it is usually machine-produced. Here is a real reconciliation, of the toponym Exeter scoped to GB, as our service returns it today:
gn:2649808 Exeter score 100
wd:Q24768 Exeter score 99
wd:Q134672 Exeter score 99
wd:Q21886091 Exeter score 99
One GeoNames record and three different Wikidata records, all plausible. A user confirms one of them; the rest may be accepted at lower confidence or left alone. In LPF today every accepted link is asserted identically, and a consumer cannot tell a human-confirmed identification from a machine guess that nobody has reviewed.
The asymmetry within the format is the part that looks accidental rather than intended: the same assertion carries a confidence if you model it as a relation, and loses it if you model it as a link.
Proposal
Permit the shared optional attributes on links[] items that relations[] items already accept — at minimum certainty, and for consistency when and citations (a match can be period-specific — two places identical in 1600 and distinct by 1900 — and a curator may want to cite the evidence for a contested identification).
"links": [
{"type": "closeMatch",
"identifier": "gn:2649808",
"certainty": "certain"},
{"type": "closeMatch",
"identifier": "wd:Q24768",
"certainty": "less-certain"}
]Purely additive: no document that is valid today becomes invalid, and consumers that don't know the attribute ignore it as they already do for relations.
2. One authoritative list of permitted identifier prefixes
Existing situation
There appear to be two lists of permitted namespace prefixes, and they disagree with each other.
- The JSON Schema we validate against (WHG-hosted, derived from LPF — see the note below) constrains
links[].identifier to either a full URL or a closed prefix set: bnf | cerl | dbp | gn | gnd | gov | indias | loc | pl | tgn | viaf | wd | wp.
- The context document defines
gn, tgn and wd as place-identifier prefixes (plus aat for types and periodo for periods). It defines none of bnf, cerl, dbp, gnd, gov, indias, loc, pl, viaf, wp — the ten the schema permits but the context does not resolve.
Two further wrinkles:
- The same CURIE is legal in one element and illegal in another.
relations[].relationTo uses the looser ^\w+:[^\s]+$, so osm:r2748782 is a valid relation target but an invalid link identifier.
- Namespaces in heavy practical use appear in neither list:
osm and ohm (OpenStreetMap / OpenHistoricalMap), po (PeriodO), and any gazetteer's own namespace for its own records. OpenStreetMap is now the single largest source of reconcilable places in WHG's index (20.6M, ahead of GeoNames at 13.5M and Wikidata at 11.5M), so a large share of the matches our tooling produces cannot be expressed as prefixed link identifiers at all. In the Exeter example above, osm:r2154556 and ohm:n2128978006 are both real candidates and neither is expressible.
Proposal
Any of these would resolve it; they're in increasing order of effort:
- Relax
links[].identifier to the same validURLorNamespaceTerm that relations[].relationTo already uses. One line, removes the internal inconsistency, and leaves prefix governance to the context document where it already lives.
- Keep a registry, but maintain it once. Define the permitted prefixes in the context document (adding at least
osm, ohm, po) and have validators derive from it rather than duplicating a closed enum.
- Document the fallback explicitly — that an authority without a registered prefix should be referenced by full URI — so producers know which way to go without reading the schema.
The honest counter-argument
A producer can always emit full URIs, which is presumably what the closed list intends. We prefer CURIEs because a single WHG export can carry millions of links, and because our own identifiers are namespace-scoped by construction (whg:<dataset_id>:<record_id>). But that preference isn't really the point: whichever way the proposal lands, two disagreeing lists — and one CURIE that validates as a relation target but not as a link identifier — look like a defect worth closing.
A question
Where does the canonical LPF JSON Schema now live? This repository holds the context document and the specification, but no schema file, and the one WHG validates against is hosted by us (https://whgazetteer.org/schema/lpf_v2.0.jsonld). If there's a newer canonical schema elsewhere, point me at it and I'll re-check both observations above against it before anyone acts on them.
What WHG has done in the meantime
So that it's on the record rather than discovered later, our local copy of the schema now deviates from the published constraints in exactly two ways, both matching the proposals above:
links[] items accept the shared optional properties (certainty, when, citations), and our exports emit certainty on every closeMatch link.- The link-identifier prefix pattern additionally admits
whg, osm, ohm and po.
We'd rather converge on whatever the working group decides than keep a local fork of the rules, and we're happy to open PRs for either or both.
Related: #47 (certainty on when), #46 (Wikidata prefix in the context document), #38 (link types).
Two proposals arising from reconciliation output:
certaintyonlinks[], and one authoritative list of identifier prefixesBoth of these come from the same use case, so I've filed them together — happy to split them if the working group would rather take them separately.
Context. World Historical Gazetteer's Map your Data workbench reconciles a user's table against gazetteers (GeoNames, Wikidata, Getty TGN, OpenStreetMap, OpenHistoricalMap, PeriodO, WHG's own contributed datasets) and exports the result as LPF, either for download or for contribution back to WHG. The output of a reconciliation is a set of
closeMatchlinks — which the README describes as "the principal means of linking places and gazetteer datasets", and for which it points at WHG's reconciliation service. Two things about that output can't currently be expressed.1. Allow
certaintyonlinks[]Existing situation
certainty—"certain"/"less-certain"/"uncertain", following the Pleiades convention — is allowed on geometries and relations, and the v1.2.2 changelog (4 September 2022) records it being extended towhen, "wherever it is used, as already allowed for the geometries and relations elements". #47, which proposed that, describes its earlier absence there as an oversight.It is not allowed on
links[].Yet a
closeMatchis exactly the kind of statement whose confidence varies, and it is usually machine-produced. Here is a real reconciliation, of the toponym Exeter scoped to GB, as our service returns it today:One GeoNames record and three different Wikidata records, all plausible. A user confirms one of them; the rest may be accepted at lower confidence or left alone. In LPF today every accepted link is asserted identically, and a consumer cannot tell a human-confirmed identification from a machine guess that nobody has reviewed.
The asymmetry within the format is the part that looks accidental rather than intended: the same assertion carries a confidence if you model it as a relation, and loses it if you model it as a link.
Proposal
Permit the shared optional attributes on
links[]items thatrelations[]items already accept — at minimumcertainty, and for consistencywhenandcitations(a match can be period-specific — two places identical in 1600 and distinct by 1900 — and a curator may want to cite the evidence for a contested identification).Purely additive: no document that is valid today becomes invalid, and consumers that don't know the attribute ignore it as they already do for relations.
2. One authoritative list of permitted identifier prefixes
Existing situation
There appear to be two lists of permitted namespace prefixes, and they disagree with each other.
links[].identifierto either a full URL or a closed prefix set:bnf | cerl | dbp | gn | gnd | gov | indias | loc | pl | tgn | viaf | wd | wp.gn,tgnandwdas place-identifier prefixes (plusaatfor types andperiodofor periods). It defines none ofbnf,cerl,dbp,gnd,gov,indias,loc,pl,viaf,wp— the ten the schema permits but the context does not resolve.Two further wrinkles:
relations[].relationTouses the looser^\w+:[^\s]+$, soosm:r2748782is a valid relation target but an invalid link identifier.osmandohm(OpenStreetMap / OpenHistoricalMap),po(PeriodO), and any gazetteer's own namespace for its own records. OpenStreetMap is now the single largest source of reconcilable places in WHG's index (20.6M, ahead of GeoNames at 13.5M and Wikidata at 11.5M), so a large share of the matches our tooling produces cannot be expressed as prefixed link identifiers at all. In the Exeter example above,osm:r2154556andohm:n2128978006are both real candidates and neither is expressible.Proposal
Any of these would resolve it; they're in increasing order of effort:
links[].identifierto the samevalidURLorNamespaceTermthatrelations[].relationToalready uses. One line, removes the internal inconsistency, and leaves prefix governance to the context document where it already lives.osm,ohm,po) and have validators derive from it rather than duplicating a closed enum.The honest counter-argument
A producer can always emit full URIs, which is presumably what the closed list intends. We prefer CURIEs because a single WHG export can carry millions of links, and because our own identifiers are namespace-scoped by construction (
whg:<dataset_id>:<record_id>). But that preference isn't really the point: whichever way the proposal lands, two disagreeing lists — and one CURIE that validates as a relation target but not as a link identifier — look like a defect worth closing.A question
Where does the canonical LPF JSON Schema now live? This repository holds the context document and the specification, but no schema file, and the one WHG validates against is hosted by us (
https://whgazetteer.org/schema/lpf_v2.0.jsonld). If there's a newer canonical schema elsewhere, point me at it and I'll re-check both observations above against it before anyone acts on them.What WHG has done in the meantime
So that it's on the record rather than discovered later, our local copy of the schema now deviates from the published constraints in exactly two ways, both matching the proposals above:
links[]items accept the shared optional properties (certainty,when,citations), and our exports emitcertaintyon everycloseMatchlink.whg,osm,ohmandpo.We'd rather converge on whatever the working group decides than keep a local fork of the rules, and we're happy to open PRs for either or both.
Related: #47 (certainty on
when), #46 (Wikidata prefix in the context document), #38 (link types).