RFC 091: Digitisation ingest identifiers during the Sierra to Folio migration

[kenoir]

Preview

View the rendered RFC on this branch (renders the Mermaid diagrams and tables):

• RFC 091 README

---

What does this change?

Publishes the digitisation-ingest discovery work as RFC 091. The RFC sets out how digitisation ingest identifiers should work across the Sierra to Folio LMS migration.

Today the Sierra b-number is the shared key for digitised content. It is the Goobi ingest package id, the storage externalIdentifier (digitised/<b-number>/vN), the METS work's own identity (mets/<b-number>), the merge candidate that joins the METS work onto the public catalogue work, and the IIIF manifest URI and DLCS asset ids. The migration removes the b-number, so the RFC explains why it cannot simply be swapped for a Folio id (doing so would reset storage versioning and silently break the merge join key, leaving digitised content invisible on the public record with no error), and proposes minting catalogue-style identifiers at ingest instead.

The proposal builds a small application for digital production staff that creates ingest package metadata from Folio, and splits behaviour on whether the Folio record carries a predecessor pointer to an old b-number:

• Cross-migration (predecessor present): reuse the old b-number as the storage handle and keep emitting the sierra-system-number merge candidate. This is the lower-risk path; it needs no transformer change and leans on RFC 083 predecessor relationships to keep canonical ids stable. After migration it needs only Folio-target merger rules.
• Post-migration (no predecessor): mint a catalogue-style id (C) via a secured resolve-or-mint endpoint added to the Identifiers API (RFC 089). C becomes the storage handle; the Folio id rides in the METS record identifier so the transformer can emit a folio-instance merge candidate and build the mets/<folio-id> own-identity. The transformer carries C from the storage externalIdentifier into the work's identifier block, and the id_minter verifies the mets/<folio-id> ↔ C binding (raising rather than minting on a mismatch). This path is blocked on RFC 081's IIdentityService in iiif-builder.

The RFC also covers alternatives considered, the merger-rule changes needed for Folio targets, dependencies for each path, and two open questions.

Files changed:

• rfcs/091-digitisation-ingest-identifiers/README.md: the RFC document (with two Mermaid sequence diagrams for the current and proposed ingest flows).
• rfcs/README.md: RFC listing table regenerated by .scripts/create_table_summary.py (RFC 091 row added).

How to test

• Read the rendered RFC and review the context, proposal, alternatives, dependencies, and open questions.
• Confirm the RFC passes repo validation: .scripts/validate_rfc.py.
• Confirm the listing table is in sync: .scripts/create_table_summary.py --check-readme.
• Some relative links (../085…/, ../088…/, ../089…/, ../090…/) resolve only once those branches merge; they are correct against the merged layout.

How can we measure success?

No measurable runtime success criteria; this is a documentation RFC. Success is the RFC being reviewed and the team aligning on the digitisation-ingest identifier approach across the migration, in particular the decision to mint catalogue-style ids rather than reuse the Folio id, the cross-migration / post-migration split, and the secured mint endpoint on RFC 089.

Have we considered potential risks?

• Documentation only; no production code or infrastructure is changed by this PR, so there is no runtime or deployment risk.
• The design risks are enumerated in the RFC and are the subject of review. The principal one is the silent merge break that follows from moving the package id off the b-number; the proposal mitigates it by reusing the b-number on the cross-migration path and by a fail-loud guard on the post-migration path (the id_minter raises rather than minting a divergent id when a METS source identifier is not already bound to its storage handle).
• Two open questions remain, both flagged as gating the post-migration path: confirming iiif-builder and DLCS make no assumption that the storage externalIdentifier (C) equals the METS record identifier (folio-id); and confirming the id_minter can verify a supplied source-id ↔ canonical-id binding and raise on a mismatch, rather than only assigning a fresh id.

Notes for reviewers

• This RFC extends RFC 089 (Identifiers API, open in RFC 089: Identifiers API #156) with a secured mint endpoint, and depends on RFC 081 (IIdentityService in iiif-builder) for the post-migration path. Related in-flight work: RFC 085 (IIIF identities, RFC 085: IIIF identities #143), RFC 088 (identity/requesting migration, RFC 088: Record the barcode sequence service #158), RFC 090 (CMS↔LMS sync, RFC 090: CMS and LMS Sync #157), and RFC 083 (stable identifiers) on main.
• Opened as a draft for discussion.

🤖 Generated with Claude Code

[tomcrane]

If I have understood correctly, the identifier defined here as C is the same concept as the identifier suggested by Paul in the on-site meeting we had a few weeks ago (described as Preservation identifier in this Slack conversation). That is, the identifier the Storage Service, Goobi and Archivematica use to name and save things, and recover old versions of things (packages in storage, workflows in Goobi). And that the RFC describes a way of ensuring that these systems always get given this identifier, which for old things may look like a b number or a Calm ref (see below), but for wholly new things will look like something else... examples will help.

I think this identifier needs a really clear name so we always know what we are talking about.

I know this is about b-numbers and FOLIO migration, but is it worth putting more examples into the RFC for the equivalent changes that have to happen for CALM Refs in both digitised and born digital content? For example in the sections

• METS works keep their own id
• Where each identifier lives after migration

And in the section

• Two kinds of digitisation

It's not just digitisation; a born digital item has the same issues, just with a different identifier form. It's really about preservation:

• Cross-migration preservation
• Post-migration preservation

Digitisation happens before preservation for physical objects, but digitisation is not a step for born digital archive material. Yet they face the same identity issues. Preservation is a precursor to all IIIF-ification, whether born digital or digitised, whether source identity is bibliographic or archival.

Unlike b numbers, hierarchical archival identifiers (AA/BB/37) still exist in the new world. So it's not quite the same, but there is one desired change that has some overlap and may have some implications for the proposal here. Our intention is that the canonical form for all IIIF representations is the Catalogue Work ID, not just those whose canonical form is currently a b number. At the moment the b number is the canonical form for all works that have b numbers, but some do not - in particular, born digital archival material from CALM, and old Miro images.

Consider:

https://iiif.wellcomecollection.org/presentation/SB/1/1/298

This redirects to https://iiif.wellcomecollection.org/presentation/b19995271, because, even though it's in CALM, it:

• has a Sierra b number, too
• It came through Goobi
• Is in storage under the digitised space - at digitised/b19995271

Its preservation identifier / externalIdentifier / C is b19995271 and therefore its current canonical identifier, as far as Goobi, Storage Service and IIIF is concerned, is b19995271. Goobi and the Storage Service will need to carry on treating this object as b19995271 forever, because it's what they currently know it as; it's reflected in METS IDs and storage location.

It has a work ID, which is dpv7wsth.

In the future, this IIIF Manifest's canonical URL will therefore be https://iiif.wellcomecollection.org/presentation/dpv7wsth

Now consider:

https://iiif.wellcomecollection.org/presentation/SABTS/A/2/10

It looks similar, but it doesn't redirect, because:

• it never had a Sierra b number
• It came through Archivematica as born digital
• It is in storage under the born-digital space, at born-digital/SABTS/A/2/10

Its preservation identifier / externalIdentifier / C is SABTS/A/2/10 and therefore its current canonical identifier, as far as Archivematica, Storage Service and IIIF is concerned, is SABTS/A/2/10. Archivematica and the Storage Service will need to carry on treating this object as SABTS/A/2/10 forever, because it's what they currently know it as; it's reflected in the storage location.

It has a work ID, which is h3jc4wga.

In the future, this IIIF Manifest's canonical URL will therefore be https://iiif.wellcomecollection.org/presentation/h3jc4wga

...we will start redirecting it if we receive the archival ref form ../SABTS/A/2/10, even though we don't now. (That's the difference between the two for IIIF delivery).

This is relevant to the PR because the notion of "preservation identifier" / C - the identifier that Goobi, Archivematica and the Storage Service see - is not just confined to b numbers. This isn't ignored in the RFC, e.g.,

"If it matches the Sierra system-number regex it emits a merge candidate sierra-system-number/<b-number>, otherwise it falls back to calm-ref-no for born-digital"

...but maybe it could be clearer and have a good name: there are CALM Ref preservation identifiers, too, that are not b numbers but continue to be the identifier the storage service understands the thing to be, forever.

I think that this is what's meant in the paragraph Canonical id continuity and in the proposal, but I find the terminology confusing (maybe not close enough to it) - different identifiers are canonical in different contexts: an old digitised object's canonical identifier is a b number as far as Goobi and Storage are concerned, and remain so forever. But the same item's canonical IIIF Manifest identifier uses the work ID.

If all Goobi or Archivematica ever rely on when processing is this preservation identifier / C, then they will continue to emit it to the SNS topic when they finish a workflow. And DDS will pick it up, and always see it as b19995271 or SABTS/A/2/10 or whatever.

Post-migration digitisation (no predecessor): mint a fresh catalogue-style id.

Can it be clearer what this catalogue-style id looks like, for new items after the migration has happened:

• Physical item catalogued in Folio, digitised through Goobi (e.g., printed book)
• Physical item catalogued in Calm/Axiell Collections, digitised through Goobi (e.g., some handwritten letters, an archival item)
• Born digital item catalogued in Calm/Axiell Collections, processed through Archivematica
• (Born digital item catalogued in Folio? That is the equivalent of the use case that triggered the DDS' own identity service, see below)

I assume this catalogue-style id is NOT the workId, but minted separately? And it's not the folio id or archival ref either.

I don't know what folio IDs look like but suppose there's an entirely new work - a printed book - being digitised with folio id fol123. It has a catalogue/workId abcd3456. We mint a C / preservation identifier with value wxyz9876, and so it gets preserved in storage at digitised/wxyz9876, and so Goobi makes a METS file called wxyz9876.xml, and all the page image JP2s are called wxyz9876_001.jp2, wxyz9876_002.jp2, etc., and Goobi fires an end-workflow message to SQS with the identifier wxyz9876 for iiif-builder (and others) to pick up.

In this scenario, the IIIF manifest that iiif-builder builds and saves to S3 should be stored as abcd3456.json not wxyz9876.json, because we want to save it in S3 as its public canonical form rather than always having to look up the "storage" form on every manifest request. If someone asks the Catalogue API for abcd3456, the digital location returned in the full API response should be ../presentation/abcd3456 and not .../presentation/wxyz9876 because the public-facing canonical IIIF URL is the semantic one not the preservation one; the work page URI on wc.org for this item has the same last path element as the IIIF Manifest, abcd3456.

On receipt of the workflow message, iiif-builder would pass "wxyz9876" to the identity service and get back a list that included blocks for fol123 and abcd3456. It knows how to select abcd3456 as the canonical identifier to publish the Manifest as.

A series of examples with identifiers for all of the digitised/born-digital, folio/axiell permutations would be really useful and may catch something lurking here.

Other random observations:

• Today, if I know the b number I know (for digitised things) where the preserved files are in storage and I know what the IIIF Manifest URL is, because they are all b numbers (assuming it's a Sierra book).
• Today, if I know the Catalogue workId, I can query the catalogue API and get back enough info to deduce the b number and hence where it lives in storage, and also the IIIF Manifest URL.
• In future, if I know the folio ID, I don't know where it's preserved or what its Manifest URI is. I query the catalogue API or identity service to learn these things.

iiif-builder prerequisite

IIdentityService (proposed in RFC 081 is implemented - it's internal to DDS. It's not fully implemented because the complete implementation depends on the external identity service.

I think it's worth having a comparison of what these two identity services do.

The main driver for the Identity Service in RFC 081 was:

[Collections would like to] ingest born-digital items through Archivematica under bnumbers from Sierra.

(This may be on hold due to the migration work).

Rather than have the DDS parse identifiers and deduce what they are (b numbers etc), where they are stored (digitised or born-digital space), what flavour of METS file they are expected to have (Goobi or Archivematica) etc., the DDS calls a service, passing the identifier (always the resolved currently canonical identifier, so currently never a Catalogue Work ID) and getting back a DdsIdentity object that carries all the info DDS needs to know.

The current implementation of this service still uses the same string parsing logic for now, but it also stores identities in a database, and validates that its conclusions about storage space and source are in fact correct. This was originally the groundwork for "bnumbers through Archivematica" - to remove the assumption that b-number means goobi, b-number means digitised storage space, etc., with the possibility that the information "this identifier's work is preserved in the born-digital storage space" is later obtained from something external to the DDS.

This implementation is not an identity resolution service as proposed in RFC 091. You can't give it a Catalogue workID and expect it to return a DdsIdentity object, because it expects the canonical form (b number or CALM Ref) and then tells you various things about it, rather than accepting any identifier form and telling you the canonical form and aliases.

It also does things that the proposed platform identity service isn't designed to do, such as understand that if you pass b12345678_002, it's a volume-level identifier for which the package-level identifier is b12345678. That is, it returns information about sub-package identifiers that the 091 Identity Service would not know.

However, the current PersistedIdentityService implementation could be extended to make calls to the proposed 091 service, so that inside the DDS, we can pass any identifier to our service and using a combination of existing logic and resolution via the 091 service.

You could say there should be one identity service to rule them all, the authoritative means of resolving an identity to something and learning about its aliases. But the distinction is that the 091 service is about the Catalogue: semantic records for works. Whereas the DDS service is about digital objects and their sub-structure. Inner structure of a digital object has no Catalogue description, therefore the 091 Identity service shouldn't be concerned with it. It's the DDS's job to be authoritative.

If it is useful for other Wellcome processes or workflows to know about these internal structure things, then the DDS should expose it as API.

[kenoir]

In this scenario, the IIIF manifest that iiif-builder builds and saves to S3 should be stored as abcd3456.json not wxyz9876.json, because we want to save it in S3 as its public canonical form rather than always having to look up the "storage" form on every manifest request. If someone asks the Catalogue API for abcd3456, the digital location returned in the full API response should be ../presentation/abcd3456 and not .../presentation/wxyz9876 because the public-facing canonical IIIF URL is the semantic one not the preservation one; the work page URI on wc.org for this item has the same last path element as the IIIF Manifest, abcd3456.

This is really useful feedback, but I want to focus in on this paragraph in particular because it will help clarify our position.

In you example above the following canonical ids exist:

• abcd3456 the LMS sourced work canonical ID, with source id like my_lms/123
• wxyz9876 the preservation system sourced work canonical ID, with source id like mets/123

In the catalogue pipeline matcher/merger we will have noticed that these refer to the same work from merge candidates, and merged them such that abcd3456 gets the digital locations from wxyz9876 and have generated a
redirection from wxyz9876 to abcd3456 such that requesting the former directs you to the latter.

In our proposal the digital locations can carry identifiers with whatever context we have at that point, it could be the METS source id mets/123, the corresponding canonical id, wxyz9876 or even the redirected target abcd3456 but I think the redirected target is the wrong choice as DDS will not have that ID available at point of ingest.

See https://github.com/wellcomecollection/docs/blob/rk/rfc-091-digitisation-ingest-identifiers/rfcs/091-digitisation-ingest-identifiers/README.md#mets-works-keep-their-own-id

The storage service will get the mets/123 mapped wxyz9876 canonical id as the externalIdentifier, with the original source identifier mets/123 inside the METS. So DDS will have that context only to decide manifest URLs.

A crucial point is that this re-direction from wxyz9876 to abcd3456 is only known at merge time based on merge candidates derived from inspecting work metadata, and that state is not fixed if we change the redirection target in the merger. The upshot of this is we cannot give preservation sourced works the canonical id of the LMS target as it breaks some of the assumptions of the pipeline.

If someone asks the Catalogue API for abcd3456, the digital location returned in the full API response should be ../presentation/abcd3456 and not .../presentation/wxyz9876 because the public-facing canonical IIIF URL is the semantic one not the preservation one

I think in this case in the IIIF APIs /presentation/wxyz9876 the preservation work canonical ID can serve as both the preservation and semantic URL.

Worth clarifying the relationship with predecessor ID which is different: We commit that it will always be true that mets/123 has the canonical ID wxyz9876, and in the case of a predecessor relationship being identified in the id minter that the successor will always get the same canonical_id as its predecessor, e.g. mets/b1234567 has wxyz9876, and also mets/folio-123 has wxyz9876. The predecessor relationship is intended to be used when migrating the source work records between systems that give them the same identifiers, works sourced from old & new systems cannot exist in the same pipeline, a fundamental difference from pipeline matching & merging.

[tomcrane]

A concrete example

Something that exists today:

https://wellcomecollection.org/works/km3uczhf is the work page
https://iiif.wellcomecollection.org/presentation/km3uczhf is the IIIF Manifest

atm the latter redirects to ../b28047345 but later it won't, we want to use the catalogue ID here.

Something created in 2027:

https://wellcomecollection.org/works/abcd3456 is the work page (if I understand correctly)

...but do we want the canonical IIIF Manifest to be
https://iiif.wellcomecollection.org/presentation/abcd3456 as well (like the above), or
https://iiif.wellcomecollection.org/presentation/wxyz9876 ?

You say:

...or even the redirected target abcd3456 but I think the redirected target is the wrong choice as DDS will not have that ID available at point of ingest.

Which implies that at initial Manifest construction time (after the Goobi workflow finishes) we'd have to build our Manifest as wxyz9876.json... if we looked up wxyz9876 at that point could we not learn that abcd3456 is its canonical ID?

That doesn't mean we can't have https://iiif.wellcomecollection.org/presentation/abcd3456 as the canonical IIIF Manifest URL, because we can resolve all these identities at the point of service, but we should decide whether that is in fact what we want. It seems neatest and user-friendliest to me that the work page and the manifest being shown on that work page have the same identifier element in them, but it is obviously not a technical necessity

[kenoir]

if we looked up wxyz9876 at that point could we not learn that abcd3456 is its canonical ID?

Yes, that's correct wxyz9876 is the canonical id of the mets sourced work, abcd3456 is the canonical id of the LMS sourced work that cannot be conflated with with the mets sourced work as the catalogue pipeline must continue to match and merge these downstream.

This could be hinting that we should not take this path and instead continue to represent METS sourced works by source identifiers in their current style, i.e. mets/123-lms-id-value. In that situation keeping the id value the same carries the benefit to digital production staff where the thing came from, rather than obscuring it, which the canonical ID would.

We could then simplify this RFC to include only the at point of ingestion tool that keeps records from the new LMS with sierra predecessor IDs on their previous system IDs.

[kenoir]

Alternatively we could keep minting the catalogue identify at point of ingestion, if we can be comfortable with the IIIF endpoints using the METS sourced work canonical identifier, to continue the example: wxyz9876, but on consideration I am leaning towards the non-opaque identifiers being useful for anyone handling these records between digitisation and preservation.

The identifiers API can still exchange these source identifiers for public catalogue identifiers, but cannot hold the result of redirection from merge, the catalogue-api is still the best place to find that.

[paul-butcher]

Yes! It looks like we should continue using source ids as storage identifiers.

I suppose there's an interesting question about who/what the identifiers are for and what they mean. I had pictured them as externally-facing so should be an external-style id.

However, that does not necessarily follow. Ids in the storage service or in METS or IIIF files or whatever are not intended to be shared manually like Wellcome Canonical Ids for Works and Concepts are. These ids could just as well be UUIDs as anything else.

Whatever id scheme we choose, it is not designed to carry meaning; but that doesn't mean it cannot be generated directly from a meaningful identifier (i.e. there's nothing wrong with saying that the id mets/123-lms-id-value is not meaningful in this context). That said, they definitely correspond to the proximal/source work, and not the result of any merge. Partly because that is where the relationship is most "real", but mainly because there is no reason why the resulting merged work would not be made up of multiple works that all have objects in the storage service.

I can imagine a scenario where matching/merging rules change meaning that where we did have a matching storage id and work id, they no longer match. It might cause a bit of a debugging headache where I try to find the storage object that matches the work id. If they never matched (because they are always in different schemes) then that confusion won't arise.