Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/cli/changelog/cmd-backfill.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ For each product in scope the command:
2. Parses each `## {version}` section into typed entries. Unrecognized `### …` subsections whose body is bullets become `Other`-typed entries; prose subsections are preserved in the bundle description.
3. Writes bundle YAML to `{output}/{product}/changelog/bundles/{version}.yaml`.
4. Writes per-entry YAML files: `{pr}.yaml` for entries that reference a pull request, `note-{slug}.yaml` for entries with no PR reference.
5. Writes `notes-{target}.json` per version listing all `note-*.yaml` files for that target (the notes registry). This is a **local** backfill artifact, not the CDN dual-write layout (`notes-{product}-{version}.json` plus legacy `notes-{version}.json`).
5. Writes `notes-{target}.json` per version listing all `note-*.yaml` files for that target (the notes registry). This is a **local** backfill artifact, not the CDN key `notes-{product}-{version}.json`.
6. Prints a per-product report including entry counts and no-PR rates.

The primary use case is measuring how much published release-notes content cannot be traced back to a PR — determining how much historical content needs the PR-less "note" format from [docs-eng-team#789](https://github.com/elastic/docs-eng-team/issues/789).
Expand Down
10 changes: 4 additions & 6 deletions docs/cli/changelog/cmd-note.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,12 +59,10 @@ products:
## After creation

Upload is the same as for other changelog YAML files.
The scrubber writes two indexes per note:

- `changelog/{org}/{repo}/notes-{product}-{version}.json` — notes for that product and version. `changelog bundle` reads this key first.
- `changelog/{org}/{repo}/notes-{version}.json` — the union of notes for that version across products, kept for older CLI pins.

`{changelog}` does not read these indexes. It loads published bundle YAML (and amend sidecars listed in `bundle/{product}/registry.json`).
The scrubber writes `changelog/{org}/{repo}/notes-{product}-{version}.json` listing every note
file for that product and version. `changelog bundle` reads that key. `{changelog}` does not —
it loads published bundle YAML (and amend sidecars listed in `bundle/{product}/registry.json`),
so removing a notes-index object from S3 does not change `:cdn:` pages.

If the release bundle for that product and version or date has already shipped when you upload, the scrubber generates an amend sidecar for **that product**, then rebuilds that product's `bundle/{product}/registry.json` so `{changelog}` `:cdn:` pages pick it up. Other products that share the version are left alone.

Expand Down
42 changes: 17 additions & 25 deletions docs/development/changelog-bundle-registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,10 +80,10 @@ narrowed reconciliation to the bundle tree):
same way as numbered `.amend-{N}` files, after those numbered amends. The `.amend-notes`
suffix is **reserved** — do not create files with that suffix manually; see
[](/cli/changelog/bundle-amend.md).
- **Notes index** — `changelog/{org}/{repo}/notes-{product}-{version}.json` (product-scoped) plus a dual-written
legacy `changelog/{org}/{repo}/notes-{version}.json` (version union), **public bucket only**, produced by
the scrubber Lambda's `NotesIndexReconciler`. See
[Notes-index format](#notes-index-format) below.
- **Notes index** — `changelog/{org}/{repo}/notes-{product}-{version}.json`, **public bucket
only**, produced by the scrubber Lambda's `NotesIndexReconciler`. Refer to
[Notes-index format](#notes-index-format). `{changelog}` `:cdn:` does not read
these keys; it loads `bundle/{product}/registry.json` and the YAML files that manifest lists.
- **Changelog-entry index** — `changelog/{org}/{repo}/{branch}/registry.json`, a **legacy
client-authored pass-through**: the current `changelog upload` never writes one, but manifests
written by older CLI versions are still mirrored verbatim from the private bucket, because
Expand Down Expand Up @@ -129,15 +129,12 @@ zero-bundle state. The reconciler deliberately restores the former.
## Notes-index format [notes-index-format]

For each product and release version that has at least one note, the scrubber Lambda writes a
product-scoped notes index at `changelog/{org}/{repo}/notes-{product}-{version}.json`. It also
dual-writes a legacy version-union index at `changelog/{org}/{repo}/notes-{version}.json` so
older `changelog bundle` clients that still GET the version-only key keep working. The product
and version segments are concatenated into the new filename; consumers must not parse that slug
apart.
product-scoped notes index at `changelog/{org}/{repo}/notes-{product}-{version}.json`. The product
and version segments are concatenated into the filename; consumers must not parse that slug apart.

`changelog bundle` GETs the product-scoped key first and falls back to the version-only key
only on HTTP 404. An empty product-scoped index does not fall back. `{changelog}` does not read
these indexes; it loads bundle YAML from the product registry.
`changelog bundle` GETs this key. A 404 means no notes for that product and version.
`{changelog}` does not read these indexes. It loads bundle YAML from the product registry
(`bundle/{product}/registry.json` and the listed parent plus amend files).

Product-scoped schema (`schema_version: 1`):

Expand All @@ -153,25 +150,21 @@ Product-scoped schema (`schema_version: 1`):
}
```

The legacy `notes-{version}.json` body omits `product` and `version` and lists the union of note
paths for that version across products.

| Field | Meaning |
|---|---|
| `schema_version` | Schema version. Currently `1`. |
| `product` / `version` | Set on product-scoped indexes; omitted from the legacy version-union body. |
| `product` / `version` | Product id and release version this index belongs to. |
| `notes[].path` | Pool-relative path of the note within `changelog/{org}/{repo}/`. The leading segment before the first `/` is the branch. |
| `notes[].bundle_seq` | Derived reporting field: `0` = no bundle published for this version yet, `1` = note shipped in the original bundle, `2` = note carried by the Lambda-generated `.amend-notes` sidecar. |

`bundle_seq` is derived — it is never authored and never a latch. The Lambda recomputes it on every
reconcile by comparing the notes index against the set of entries in the published bundle and its
amend sidecars.

A 404 on both the product-scoped index and the legacy version-union index means "no notes
published for this product and version". An empty `notes` array never appears on a successfully
reconciled index — the index is deleted rather than emptied, following the same
[absent ≠ empty](#absent-empty) rule as the bundle registry. Until older clients stop reading
`notes-{version}.json`, the Lambda keeps that key while any note still declares the version.
A 404 on the product-scoped index means "no notes published for this product and version". An
empty `notes` array never appears on a successfully reconciled index — the index is deleted
rather than emptied, following the same [absent ≠ empty](#absent-empty) rule as the bundle
registry.

## Shallow per-tree change maps [shallow-maps]

Expand Down Expand Up @@ -322,10 +315,8 @@ ship silently). `CdnChangelogEntryFetcher` reuses a shared `HttpClient` in produ
owned client only when a test injects a handler, mirroring `CdnChangelogFetcher`.

When CDN sourcing is on, the same fetcher also merges changelog notes for each concrete
`--output-products` pair: it GETs `changelog/{org}/{repo}/notes-{product}-{version}.json` first
and falls back to `notes-{version}.json` only on HTTP 404. See
[Notes-index format](#notes-index-format). Checksum dedup keeps a single copy if two products at
the same version still share the legacy index.
`--output-products` pair: it GETs `changelog/{org}/{repo}/notes-{product}-{version}.json`. See
[Notes-index format](#notes-index-format). `{changelog}` `:cdn:` does not use this path.

## Consumer: `{changelog}` directive `cdn:` mode (implemented)

Expand Down Expand Up @@ -382,6 +373,7 @@ notes use `NoopReleaseNotesResolver`.
2. Parse it; for each `bundles[].file`, `GET {cdnBase}/bundle/{product}/{file}`.
3. Feed the downloaded YAML into the existing `BundleLoader` → `MergeBundlesByTarget` →
render pipeline. **Rendering is unchanged**; only the source of the bundle bytes differs.
Notes-index objects under `changelog/{org}/{repo}/notes-*.json` are never fetched here.

Implemented by `CdnChangelogFetcher` (a stateless async fetch engine in
`Elastic.Documentation.Configuration`) and `BundleLoader.LoadBundlesFromContent`. Because public
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -489,10 +489,9 @@ private static Uri CombineSegments(Uri baseUri, IReadOnlyList<string> segments)
/// <summary>
/// Fetches all <c>note-*.yml</c> entries for <paramref name="org"/>/<paramref name="repo"/> and
/// <paramref name="product"/> at <paramref name="version"/> from the CDN. Reads
/// <c>notes-{product}-{version}.json</c> first; on HTTP 404 only, falls back to the legacy
/// <c>notes-{version}.json</c>. A missing index (both 404) means no notes (not an error).
/// An empty product-scoped index does not fall back. A listed note that cannot be fetched
/// is a hard error — the index is an authoritative promise that the note exists.
/// <c>notes-{product}-{version}.json</c>. A missing index (404) means no notes (not an error).
/// A listed note that cannot be fetched is a hard error — the index is an authoritative
/// promise that the note exists.
/// </summary>
public async Task<IReadOnlyList<CdnChangelogEntry>> FetchNotesAsync(
Uri baseUri,
Expand All @@ -516,44 +515,26 @@ Cancel ctx
return [];
}

var productIndexUri = CombineSegments(baseUri, ["changelog", org, repo, $"notes-{product}-{version}.json"]);
var (productMissing, productIndex) = await TryLoadNotesIndexAsync(
productIndexUri,
org,
repo,
version,
emitError,
ctx
).ConfigureAwait(false);
if (productIndex is not null)
return await FetchListedNotesAsync(baseUri, org, repo, version, productIndex, emitError, ctx).ConfigureAwait(false);
if (!productMissing)
return [];

var legacyIndexUri = CombineSegments(baseUri, ["changelog", org, repo, $"notes-{version}.json"]);
var (legacyMissing, legacyIndex) = await TryLoadNotesIndexAsync(legacyIndexUri, org, repo, version, emitError, ctx).ConfigureAwait(
false
);
if (legacyIndex is not null)
return await FetchListedNotesAsync(baseUri, org, repo, version, legacyIndex, emitError, ctx).ConfigureAwait(false);
if (legacyMissing)
var indexUri = CombineSegments(baseUri, ["changelog", org, repo, $"notes-{product}-{version}.json"]);
var (missing, index) = await TryLoadNotesIndexAsync(indexUri, org, repo, version, emitError, ctx).ConfigureAwait(false);
if (index is not null)
return await FetchListedNotesAsync(baseUri, org, repo, version, index, emitError, ctx).ConfigureAwait(false);
if (missing)
{
_logger.LogDebug(
"Notes index for {Org}/{Repo}/{Product}@{Version} not found at {ProductUri} or {LegacyUri}; no notes to bundle",
"Notes index for {Org}/{Repo}/{Product}@{Version} not found at {Uri}; no notes to bundle",
org,
repo,
product,
version,
productIndexUri,
legacyIndexUri
indexUri
);
}
return [];
}

/// <summary>
/// Loads a notes index. Returns <c>missing: true</c> only on HTTP 404. A present but empty
/// or unparseable body is <c>missing: false</c> with a null index so callers do not fall back.
/// Loads a notes index. Returns <c>missing: true</c> only on HTTP 404.
/// </summary>
private async Task<(bool Missing, NotesIndex? Index)> TryLoadNotesIndexAsync(
Uri indexUri,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ namespace Elastic.Documentation.Configuration.ReleaseNotes;
/// Fetches changelog bundles for a single product from the public CDN. It reads
/// <c>{base}/bundle/{product}/registry.json</c> to enumerate bundles, downloads each
/// <c>{base}/bundle/{product}/{file}</c>, and parses them via
/// <see cref="BundleLoader.LoadBundlesFromContent"/>.
/// <see cref="BundleLoader.LoadBundlesFromContent"/>. It does not read notes indexes
/// (<c>changelog/{org}/{repo}/notes-*.json</c>); those are used only by <c>changelog bundle</c>.
/// </summary>
/// <remarks>
/// <para>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,17 +93,9 @@ public static string ChangelogFileKey(string org, string repo, string branch, st
public static string ChangelogRegistryKey(string poolGroup) => $"{ChangelogPrefix}{poolGroup}/{RegistryFileName}";

/// <summary>
/// The legacy notes-index key for one release version within a repo:
/// <c>changelog/{org}/{repo}/notes-{version}.json</c>.
/// Repo-level and branch-agnostic — the union of notes for that version across every product.
/// The leftover version-only key <c>changelog/{org}/{repo}/notes-{version}.json</c>.
/// No longer written. The notes reconciler deletes remaining objects at this key.
/// </summary>
/// <remarks>
/// The slug in the key is the release version (e.g. <c>9.3.0</c> or <c>2026-05-15</c>).
/// Previously this parameter was named <c>target</c> to match the obsolete <c>target:</c> YAML field;
/// it was renamed to <c>version</c> when that field was replaced by <c>versions:</c> on notes.
/// Older <c>changelog bundle</c> clients still GET this key; the scrubber dual-writes it
/// alongside <see cref="NotesIndexKey(string, string, string, string)"/>.
/// </remarks>
public static string NotesIndexKey(string org, string repo, string version) => $"{ChangelogPrefix}{org}/{repo}/notes-{version}.json";

/// <summary>
Expand All @@ -123,8 +115,9 @@ public static string NotesIndexKey(string org, string repo, string product, stri
/// <summary>
/// Returns true when <paramref name="key"/> is a notes-index key of the form
/// <c>changelog/{org}/{repo}/notes-{slug}.json</c> (exactly two group segments, then
/// a <c>notes-</c>-prefixed JSON file with a non-empty slug). Covers both the legacy
/// <c>notes-{version}.json</c> shape and the product-scoped <c>notes-{product}-{version}.json</c> shape.
/// a <c>notes-</c>-prefixed JSON file with a non-empty slug). Covers the product-scoped
/// <c>notes-{product}-{version}.json</c> shape and leftover version-only
/// <c>notes-{version}.json</c> objects so they can still be listed and deleted.
/// </summary>
public static bool IsNotesIndex(string key)
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,17 +41,15 @@ public sealed record NoteIndexEntry
}

/// <summary>
/// Notes index published at <c>changelog/{org}/{repo}/notes-{product}-{version}.json</c>
/// (and dual-written to the legacy <c>notes-{version}.json</c> key).
/// Notes index published at <c>changelog/{org}/{repo}/notes-{product}-{version}.json</c>.
/// Lists all <c>note-*.yml</c> fragments for one product and release version,
/// across every branch of the repo.
/// </summary>
/// <remarks>
/// Contents are paths, not bodies — the note files remain the single source of truth.
/// A stale index can only omit or over-list, never serve stale prose. Bundling a version
/// is therefore 1 GET for the index + one GET per listed note.
/// <see cref="Product"/> and <see cref="Version"/> are set on product-scoped writes and
/// omitted from legacy version-union bodies so older deserializers keep working.
/// <see cref="Product"/> and <see cref="Version"/> are set on every write.
/// </remarks>
public sealed record NotesIndex
{
Expand All @@ -62,14 +60,12 @@ public sealed record NotesIndex
public const int CurrentSchemaVersion = 1;

/// <summary>
/// Bundle product id this index belongs to. Present on product-scoped keys; omitted on
/// the legacy version-union body.
/// Bundle product id this index belongs to.
/// </summary>
public string? Product { get; init; }

/// <summary>
/// Release version this index belongs to. Present on product-scoped keys; omitted on
/// the legacy version-union body.
/// Release version this index belongs to.
/// </summary>
public string? Version { get; init; }

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1604,8 +1604,6 @@ Cancel ctx
return entries;

// Dedup by checksum: a note body identical to a PR entry (edge case) should appear once.
// Two output products at the same version may both fall back to notes-{version}.json
// until product-scoped indexes exist; checksum skip keeps a single copy.
var seen = new HashSet<string>(entries.Select(e => e.Checksum), StringComparer.OrdinalIgnoreCase);
var combined = new List<MatchedChangelogFile>(entries);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ namespace Elastic.Changelog.Reconciliation;
/// (<c>{parent}.amend-notes.yaml</c>) that carries notes that arrived after the release shipped.
/// Also updates each <c>bundle_seq</c> on the product-scoped notes index: 0 = no bundle yet,
/// 1 = shipped in the original bundle or a human amend, 2 = carried by the reconciler amend sidecar.
/// The dual-written legacy <c>notes-{version}.json</c> path list is refreshed in the same pass.
/// </summary>
/// <remarks>
/// <para>
Expand Down Expand Up @@ -52,8 +51,8 @@ public sealed class NoteAmendReconciler(
/// <summary>
/// For the given repository scope, scans each product that appears in
/// <paramref name="notesByProduct"/> (not every <c>bundle/{product}/</c> prefix), writes or
/// deletes that product's reconciler-owned amend sidecars, and re-writes product-scoped and
/// legacy notes indexes with correct <c>bundle_seq</c> values. Empty version lists mean that
/// deletes that product's reconciler-owned amend sidecars, and re-writes product-scoped
/// notes indexes with correct <c>bundle_seq</c> values. Empty version lists mean that
/// product×version just vanished from the notes index; amend still runs so sidecars can drop.
/// </summary>
/// <returns>
Expand Down Expand Up @@ -128,18 +127,6 @@ Cancel ctx
var updatedEntries = WithSeqs(write.Notes, seqs);
await notesIndexReconciler.WriteIndexAsync(indexKey, updatedEntries, ct, new NotesIndexMetadata(write.Product, write.Version));
});

foreach (var (version, unionNotes) in UnionByVersion(notesByProduct))
{
ctx.ThrowIfCancellationRequested();
if (unionNotes.Count == 0)
continue;

var seqs = MaxSeqsForVersion(notesByProduct.Keys, version, seqMap);
var updatedEntries = WithSeqs(unionNotes, seqs);
var indexKey = ChangelogKeys.NotesIndexKey(org, repo, version);
await notesIndexReconciler.WriteIndexAsync(indexKey, updatedEntries, ctx);
}
}

private static List<NoteIndexEntry> WithSeqs(IReadOnlyList<NoteIndexEntry> notes, IReadOnlyDictionary<string, int> seqs) =>
Expand All @@ -148,47 +135,6 @@ private static List<NoteIndexEntry> WithSeqs(IReadOnlyList<NoteIndexEntry> notes
.OrderBy(n => n.Path, StringComparer.Ordinal)
.ToList();

private static Dictionary<string, List<NoteIndexEntry>> UnionByVersion(
IReadOnlyDictionary<string, IReadOnlyDictionary<string, IReadOnlyList<NoteIndexEntry>>> notesByProduct
)
{
var byVersion = new Dictionary<string, List<NoteIndexEntry>>(StringComparer.Ordinal);
foreach (var byProductVersion in notesByProduct.Values)
{
foreach (var (version, notes) in byProductVersion)
{
if (!byVersion.TryGetValue(version, out var union))
byVersion[version] = union = [];
foreach (var note in notes)
{
if (!union.Any(e => e.Path == note.Path))
union.Add(note);
}
}
}
return byVersion;
}

private static Dictionary<string, int> MaxSeqsForVersion(
IEnumerable<string> products,
string version,
IReadOnlyDictionary<string, Dictionary<string, int>> seqMap
)
{
var max = new Dictionary<string, int>(StringComparer.Ordinal);
foreach (var product in products)
{
if (!seqMap.TryGetValue(ProductVersionKey(product, version), out var seqs))
continue;
foreach (var (path, seq) in seqs)
{
if (!max.TryGetValue(path, out var current) || seq > current)
max[path] = seq;
}
}
return max;
}

private static string ProductVersionKey(string product, string version) => $"{product}/{version}";

private async Task<bool> ProcessProductAsync(
Expand Down
Loading
Loading