Syndication failing to trigger

[gerwitz]

Description of the bug

In my configuration, posts generated via the Micropub endpoint are not syndicating until I use the UI to visit them and press the "Syndicate" button.

It works when I do this, which I believe confirms that my configuration and environment variables are correct. And also that the Micropub client (a Drafts script) is correctly setting the syndicate-to property.

I haven't yet set up a dev environment to diagnose; I'm hoping this is a known problem and I'm overlooking some magic "autoSyndicate" setting 😆

[paulrobertlloyd]

Syndication isn’t automatic, it needs to be triggered. This can be done via the UI, or by pinging the /syndicate endpoint. I’ve given up with the latter approach as more often than not it ends up syndicating the wrong posts, or syndicating a post multiple times.

As it stands, I think the current approach is an architectural dead-end. I’d like to move towards an event-driven model, such that a syndicator plugin can hook in to a pre-publish event, syndicate a post and add the returned syndication URL to a post’s meta data prior to publishing. That would require less round trips, and hopefully be a little less error prone.

Doing this requires a fundamental reworking of the plugin system. I made a start on this work earlier this year, but most of my time has been taken up by work and work-adjacent projects, sadly. Hopefully I can return to this soon.

[sentience]

Sorry to hear the current approach is on the way out; I use the recommended Netlify webhook approach and really value the automation of syndication.

That said, I've definitely had an occasion or two of a post being syndicated more than once. But I still don't understand what about the current approach is causing that. Is it a race condition that could be solved by some concurrency solution?

I'm not sure I fully follow your description of the event-driven model you're moving towards, but it sounds like it could cause a post to be syndicated before it appears on my site (or, in the event my site build failed for some reason, without the post being published on my site at all), which seems like a downgrade from the current version where the publication occurs before the syndication.

It's not a deal-breaker for me, but I do think I slightly prefer the current approach, if the issue of its unreliability can be resolved.

[paulrobertlloyd]

Nothing has been built and so nothing decided yet, but I think adding event hooks would offer more flexibility and predictability throughout the system.

Syndication is one of a class of architectural issues with Indiekit where we need every part of the system to work, and if one part errors, everything then falls out of sync. For syndication, each of these activities needs to occur:

• Creating a new post in the database
• Creating a new file in the content store
• Retreiving a syndicated URL
• Updating the post in the database
• Updating the file in the content store

If syndication was to occur on a pre-publish event, it would mean for a brief period a syndicated post would contain a link to a post that wasn’t published yet, but it would also mean there was less opportunities for failure (you’d only need to write a post to the database and content store once).

That said, perhaps a syndicator plug-in could be written such that if you want to include a link back to a syndicated post, syndication would then happen on a post-publish event, as it does now. That’s certainly a distinct possibility (the Internet Archive syndicator would always need to use post-publish hook, for example).

I’m hoping the time I’ve taken away from Indiekit will help me reassess the current design and approach. I want Indiekit to become easier to use and less error-prone while at the same time have an underlying architecture this is more robust and easier to reason with. Undecided if event hooks would address those goals.

Race conditions, and separate stores of data falling out of sync are persistent pain points, so if you have any other suggestions please let me know!