fix(endpoint-posts,endpoint-files): store a UUIDv7 uid and look posts up by it by rmdes · Pull Request #964 · getindiekit/indiekit

rmdes · 2026-09-13T10:03:13Z

Gives every post and media item a stored identifier — a UUIDv7 in properties.uid — and lets Micropub resolve a single post or file by it.

Fixes #924.

Following on from #925 and your suggestion there: UUIDv7 keeps the date-sortability ObjectId was providing, and is a plain string in any backend. This supersedes #925, which fixed the 404 by reading the posts collection directly — the thing you were trying to avoid. I'll close it in favour of this unless you'd rather I rebase it.

What changes

properties.uid has never been stored. It was synthesised at read time from the Mongo _id (endpoint-micropub/lib/mf2.js), and media did the same from its own _id. This stores it instead:

New posts and media get crypto.randomUUIDv7() on create.
Existing documents get one from a backfill that runs at startup, before the server accepts requests.
The read paths return the stored value, which deletes jf2ToMf2's shouldIncludeObjectId parameter — it existed only to stop the synthesised value leaking into nested vocabularies.
q=source accepts a uid alongside url, on both the Micropub and media endpoints, and the posts and files interfaces ask for the one item they want.

#924 was caused by getPostProperties requesting q=source with no url and no limit, taking getCursor's default page of 40, and searching it in memory. Anything below the newest 40 was reported missing.

getFileProperties had the same bug against the media endpoint, so a file older than the newest 40 uploads 404s from /files/:uid. It is fixed here too rather than left behind: it is the same change, and fixing only half would have left the two halves inconsistent for no reason.

Decisions you may want to push back on

q=source&uid= is an extension. The specification defines q=source with url only. uid is the identifier the posts interface holds, and resolving by it is what removes the database dependency from that route, but it is not a spec'd query.

An unknown uid returns 404, an unknown url still returns 400. endpoint.get throws on any error response, so a 400 would make the posts interface render an error page rather than its own not-found page. NotFoundError.record already exists in every locale, so this adds no strings.

The backfill is an unbounded write pass on the first boot after upgrade. It walks the collection and writes to documents that lack a uid. I have not added batching or deferral because I did not want to invent a policy you had not asked for — happy to add either if you would prefer it.

Pagination still sorts and ranges on _id. Removing that is the follow-up this PR makes possible; doing both at once would mean paginating on a field that is only half populated while the backfill runs.

uid is kept out of generated post files. It reaches the post template like any other property, so it would have appeared in the front matter of every file written to a user's content repository. It is stripped alongside post-type. Say the word if you would rather it were written — an identifier in the file is a defensible thing to want, it just should not arrive unannounced.

The migration

crypto.randomUUIDv7() always stamps the current time, so it cannot give an existing post an identifier that sorts by when the post was created. The backfill therefore carries a small generator that takes the timestamp it is given.

ObjectId.getTimestamp() resolves only to the second, while _id order within a second is well defined by its counter. Using the timestamp alone would let UUIDv7's random bits decide the order of same-second posts, which a bulk import produces by the thousand. The sequence field holds 4096 values per millisecond, so documents sharing a second spread across the milliseconds within it — 4,096,000 per second before the second is exhausted.

The backfill iterates the whole collection rather than only the un-migrated part, so the numbering is a function of the collection's contents and an interrupted run resumes to the same result. It writes only to documents that still lack a uid, so two processes starting against one database cannot overwrite each other. A document whose _id is not an ObjectId is skipped with a warning rather than failing startup; Indiekit never writes such an id itself.

No new dependency.

Testing

Full suite passes. New coverage: the generator's bit layout and ordering including the 4096-per-millisecond boundary; backfill ordering, resumability and concurrency; uid stability when a post is written to a URL that already has one; a client-supplied uid being ignored on create and on update; q=source&uid= and its 404 on both endpoints; getPostProperties and getFileProperties fetching an item outside the first page, each failing without its fix.

Also exercised as a real upgrade rather than only in fixtures. A database written by the previous version, carrying posts and media created through the interface, was restarted on this branch. The backfill gave every existing document a uid before the server bound its port, recovered each one's original creation time from its ObjectId to the second, and the resulting order matched _id order. Posts created afterwards sort correctly among the migrated ones.

Through the interface on that upgraded database: creating a post, deleting two, undeleting them, and listing throughout. A post's uid is unchanged across delete and undelete, deleted posts leave the published site and return with the undelete, and no generated file gained a uid key — including the two the undelete rewrote.

paulrobertlloyd

This looks great! Mostly nitpicking style-related changes needed.

paulrobertlloyd · 2026-09-13T23:20:48Z

+    // below, before the real uid is assigned — skewing today's count for
+    // this post type and letting a client influence another post's
+    // numbering.
+    delete properties.uid;


Is this to prevent clients sending their own UID (such as https://example.com/posts/123) which we don’t want as we prefer UUIDv7 as it’s sortable? And in postTypeCount.get() we compare on uid?

Yes to both. The delete is there because postTypeCount runs inside renderPath before the server assigns the real uid, and it excludes the current post by properties.uid, so a client-chosen value would skew the count for its type, or point it at another post. Sortability is why the server’s own value is a UUIDv7, not why the client’s is ignored.

paulrobertlloyd · 2026-09-13T23:30:03Z

Can we rebase 69851a1 into an earlier commit? (Appreciate it relates to a later change I made and merged into main).

paulrobertlloyd · 2026-09-13T23:32:19Z

One outstanding question; how long do the migration scripts remain in the codebase? Until v1.0.0? A later beta? Maybe that’s something to think about when we hopefully replace MongoDB with SQLite?

Either way, perhaps we should comment migration related code with TODO/@todo?

paulrobertlloyd · 2026-09-13T23:58:29Z

I wonder if this is worth running by the folk on the Indie Web chat? If merged, this should also be documented over at https://github.com/indieweb/micropub-extensions

Posts and media created before `properties.uid` existed have no identifier to look them up by, and their creation time survives only in the ObjectId. Walk a collection in `_id` order and assign each document a UUIDv7 whose timestamp comes from that ObjectId, so the new identifiers sort the same way the old ones did. `crypto.randomUUIDv7()` always stamps the current time, so the generator here takes the timestamp it is given. ObjectId resolves only to the second, while the sequence holds 4096 values per millisecond, so documents sharing a second spread across the milliseconds within it. Iterating the whole collection rather than only the un-migrated part keeps the numbering a function of the collection's contents, so an interrupted run resumes to the same result.

`uid` was invented at read time from the Mongo `_id`, which tied every route that keys on it to the database and left third-party Micropub servers unable to supply one. Store it instead, as a UUIDv7: a plain string in any backend that still sorts by creation time. An existing post keeps the uid it already has when a post is written to the same URL, since that value appears in Micropub responses and keys the posts interface. A uid supplied by the client is ignored; honouring it would let a client claim another post's identifier, and it reaches the post-type count through `renderPath` before the real one is assigned. `postTypeCount` compares `properties.uid` as a string rather than casting it to an ObjectId, which would throw on anything that is not 24 hex characters.

Documents written before `properties.uid` existed have no identifier, and a half-migrated collection would answer some lookups and return not found for others. Run the backfill after the plugins have registered their collections and before the server accepts requests, so every document has one by the time anything can ask. The collections are named rather than taken from `collections`, because the update creates a `properties` subdocument and would otherwise reach every collection a plugin happens to register.

Now that `uid` is a property like any other, `jf2ToMf2`'s generic loop copies it and the special case that synthesised it from `_id` has nothing left to do. The `shouldIncludeObjectId` parameter existed only to stop that synthesised value leaking into nested vocabularies, so it goes with it. `getMediaProperties` returned the raw `_id` for the same reason and now returns the stored value too.

Reading a post asked the Micropub endpoint for `q=source` with no url and no limit, took the default page of 40, and searched it in memory, so every post below the newest 40 was reported as missing. `q=source` now accepts a `uid` alongside `url` and resolves that one post through the same response path. This extends the query: the specification defines `q=source` with `url` only, and `uid` is the only identifier the posts interface holds. A url that matches nothing stays a bad request; a uid that matches nothing is a record that is gone, so it returns not found and the posts interface renders its own page for it rather than an error. Fixes #924

Deleting strips every property except those needed to rebuild the path, which was harmless while `uid` was derived from the `_id` a deleted document keeps. Nothing derives it now, so the listing had no identifier to build a link from and failed, leaving the post unreachable and so impossible to undelete.

Creating a post ignores a uid sent by the client, but updating one did not, so an update could move a post onto another post's identifier or onto a value that breaks the address the posts interface already holds. Keep the stored uid whatever the operation asks for.

Nothing exercised `getPostProperties`, so the fix for reading a post older than the default page of 40 had no test that fails without it.

It returns false when a post is not found, which is what its caller reads to raise its own not-found page.

A run that died partway restarted the sequence within a second that was already partly assigned, so those documents sorted before uids the earlier run had written, breaking the ordering the generator exists to preserve. Counting every document rather than only the un-migrated ones makes the numbering a function of the collection, so a resumed run reproduces it. Two processes starting against one database could also both write, the second overwriting a uid the first may already have served. The update now matches only documents that still lack one. A document whose `_id` is not an ObjectId is skipped rather than failing the whole startup: Indiekit never writes such an id itself, so only imported data can carry one.

The backfill reported only when it changed something, so a run that found nothing left no sign it had happened at all.

getFileProperties asked the media endpoint for an unfiltered q=source listing and scanned the default page of 40 for a matching uid, so a file older than the newest 40 uploads 404s from /files/:uid. It now asks endpoint-media for that one file by uid directly, mirroring the posts fix for the same bug (#924).

rmdes · 2026-09-14T06:17:29Z

Rebased on main with the typecheck commit folded into the three it belonged to, and the nits applied in place. Took your offer on uid in templates: the commit stripping it is gone, so it now reaches the post template like any other property.

On the migration: it should stay until every existing database has run it once, which in practice means v1.0.0 or the move off MongoDB, whichever comes first; a fresh install never needs it. Both the backfill and its call now carry an @todo saying so.

Agreed on the chat. I’ll ask this week and open a micropub-extensions issue once this merges.

rmdes force-pushed the feat/uuidv7-post-uid branch from 99e2302 to 69851a1 Compare September 13, 2026 10:27

paulrobertlloyd requested changes Sep 13, 2026

View reviewed changes

paulrobertlloyd added plugin-endpoint Endpoint plug-in core Core indiekit package labels Sep 13, 2026

paulrobertlloyd assigned rmdes Sep 13, 2026

paulrobertlloyd added this to the v1.0 milestone Sep 13, 2026

rmdes added 12 commits September 14, 2026 08:10

test(endpoint-posts): cover fetching a post outside the first page

52fbc34

Nothing exercised `getPostProperties`, so the fix for reading a post older than the default page of 40 had no test that fails without it.

docs(endpoint-posts): correct getPostProperties return type

9d32b58

It returns false when a post is not found, which is what its caller reads to raise its own not-found page.

fix(indiekit): trace the uid backfill even when it's a no-op

4ee93f4

The backfill reported only when it changed something, so a run that found nothing left no sign it had happened at all.

rmdes force-pushed the feat/uuidv7-post-uid branch from 69851a1 to 06bc7cc Compare September 14, 2026 06:12

Navigation Menu

Uh oh!

fix(endpoint-posts,endpoint-files): store a UUIDv7 uid and look posts up by it - #964

fix(endpoint-posts,endpoint-files): store a UUIDv7 uid and look posts up by it#964
rmdes wants to merge 12 commits into
mainfrom
feat/uuidv7-post-uid

rmdes commented Sep 13, 2026

Uh oh!

paulrobertlloyd left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paulrobertlloyd Sep 13, 2026

Uh oh!

rmdes Sep 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paulrobertlloyd commented Sep 13, 2026

Uh oh!

paulrobertlloyd commented Sep 13, 2026

Uh oh!

paulrobertlloyd commented Sep 13, 2026

Uh oh!

rmdes commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rmdes commented Sep 13, 2026

What changes

Decisions you may want to push back on

The migration

Testing

Uh oh!

paulrobertlloyd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paulrobertlloyd Sep 13, 2026

Choose a reason for hiding this comment

Uh oh!

rmdes Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paulrobertlloyd commented Sep 13, 2026

Uh oh!

paulrobertlloyd commented Sep 13, 2026

Uh oh!

paulrobertlloyd commented Sep 13, 2026

Uh oh!

rmdes commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants