Late changelog notes on S3 never appear on CDN changelog pages

[lcawl]

A changelog note for Elastic Cloud Enterprise 4.2.0 landed in S3 but never appeared on {changelog} pages that use :cdn:.

What happened?

I created a changelog note for Cloud Enterprise 4.2.0 after the parent bundle had already shipped. The scrubber wrote bundle/cloud-enterprise/cloud-4.2.0.amend-notes.yaml in the public bucket elastic-docs-v3-changelog-bundles.

The ECE release-notes page already uses :cdn: cloud-enterprise. docset.yml already lists that product under release_notes:. I expected the note to be published here: https://www.elastic.co/docs/release-notes/cloud-enterprise

AI analysis

I think the problem is that the {changelog} directive does not list objects in S3. At startup it only fetches files named in the product registry. The live registry in this case (generated_at: 2026-08-25T13:56:49.9548117+00:00) lists cloud-4.2.0.yaml, cloud-4.1.1.yaml, cloud-4.1.0.yaml, and cloud-ui-4.1.0.yaml. It does not list cloud-4.2.0.amend-notes.yaml.

ScrubberProcessor.ProcessAsync rebuilds bundle/{product}/registry.json only for private-bucket bundle keys. A note upload is a changelog-pool key, so that rebuild never runs. NoteAmendReconciler then writes {parent}.amend-notes.yaml to the public bucket. Public-bucket writes do not retrigger the Lambda. The sidecar can exist while the registry stays stale. The shallow map bundle/registry.json is also patched before that write.

NoteAmendReconcilerTests.LateNote_NoBundleAmendYet_WritesAmendNotesSidecar asserts the sidecar write. It does not assert that registry.json lists the file.

A second bug remains after the registry lists the sidecar. BundleAmendMerger treats .amend-notes.yaml as an amend sidecar. BundleLoader.GetParentBundlePath uses a private regex \.amend-\d+\.ya?ml$ that only matches numbered amends such as .amend-1.yaml. That regex does not match .amend-notes.yaml. Parent resolution then treats cloud-4.2.0.amend-notes.yaml as its own parent. The loader does not merge the sidecar into cloud-4.2.0.yaml. BundleLoaderFromContentTests only covers .amend-1.yaml.

Default :type: still hides known-issue, breaking-change, and deprecation. Check the note type after discovery works.

The scrubber should rebuild the product registry and the shallow map after it writes or deletes an amend-notes sidecar. BundleLoader should reuse BundleAmendMerger.GetParentBundlePath.

Related, not a duplicate: elastic/docs-builder#4072 is about unpublishing a note that already shipped.

How can we reproduce it?

1. Publish a parent bundle so bundle/{product}/registry.json lists it. Here that file is cloud-4.2.0.yaml for cloud-enterprise.
2. After that bundle exists, run docs-builder changelog note for the same product and version. Upload so the scrubber runs.
3. Confirm the public object bundle/{product}/{parent}.amend-notes.yaml exists.
4. Fetch the product registry from the CDN. The sidecar file name is absent.
5. Serve a docset whose {changelog} block uses :cdn: for that product. The note does not appear.

Any private-bucket event under bundle/{product}/ rebuilds the registry from the public listing. That would pick up a sidecar that is already there. It does not fix the next note upload.

Version or commit

Observed against docs-builder from source at deff47cd7aa878e6c6d2507735644d72fc104f7d while serving elastic/cloud. The live CDN registry is dated 2026-08-25.

Additional notes

I believe that this issue will implement changes that overlap with #4103 so the fixes will need to be coordinated

[lcawl]

Per AI analysis, a possible solution is as follows:

Recommended: Option A + Option D together

Ship both. A without D still leaves notes unmerged. D without A never downloads the sidecar.

A. Rebuild the product registry after amend-notes writes (primary)

Keep the registry as the only CDN catalog. After NoteAmendReconciler.ReconcileAsync, rebuild registry.json (and the bundle shallow map) for products whose sidecar was written, deleted, or already present and skipped as unchanged.

Where: ScrubberProcessor.ProcessAsync after the notesWork loop, before the existing shallow-map loop.

How:

• Change ReconcileAsync to return the product ids it touched (including skip-unchanged). Skip-unchanged still needs a registry pass so a sidecar written earlier can be listed.
• For each product, ChangelogScope.TryCreateBundle then BundleRegistryReconciler.ReconcileGroupAsync.
• Add those bundle scopes to shallowWork so ShallowRegistryReconciler sees the new object. Today the shallow pass runs on the pre-sidecar listing.

Do not enable public-bucket S3 events to “retrigger” the Lambda. Events come from the private bucket on purpose; a public-bucket loop is easy to get wrong.

Tests: extend ScrubberProcessorTests: seed a parent bundle + registry, process a note-*.yml private-bucket event, assert public registry.json lists {parent}.amend-notes.yaml. Extend NoteAmendReconcilerTests only if the return value needs unit coverage.

Existing drift (ECE 4.2.0): this fix does not rewrite cloud-enterprise until the next note/bundle event for that product. One operational pass: any private-bucket object under bundle/cloud-enterprise/ (or a one-off reconcile job) rebuilds the registry from the public listing. Do that after the Lambda ships, with an explicit human decision.

D. Make BundleLoader use BundleAmendMerger parent paths (secondary)

In BundleLoader, delete the private \.amend-\d+\.ya?ml$ regex. Call BundleAmendMerger.GetParentBundlePath and join with the directory. Update the MergeAmendFiles comment that still says {baseName}.amend-{N}.yaml only.

Tests: add a .amend-notes.yaml case next to LoadBundlesFromContent_AmendFile_IsMergedIntoParent in BundleLoaderFromContentTests.cs.

Alternative options that I've omitted from this post included (B) Reorder ProcessAsync only (notes, then registry, then shallow), (C) Consumer invents sidecar names, (E) Fold late notes into the parent bundle YAML, and (F) complicated operational workarounds

[lcawl]

This issue is incorporated in the fixes described in #4103 (comment)