IPIP-550: PBNode field ordering

src/ipips/ipip-0499.md

@@ -91,6 +91,7 @@ The following [UnixFS](https://specs.ipfs.tech/unixfs/) parameters were identifi
 1. [Symlink](https://specs.ipfs.tech/unixfs/#dag-pb-symlink) handling: preserved as UnixFS Type=4 nodes, or followed (dereferenced to target).
 1. [Mode](https://specs.ipfs.tech/unixfs/#mode-field): optional POSIX file permissions.
 1. [Mtime](https://specs.ipfs.tech/unixfs/#mtime-field): optional modification timestamp.
+1. [PBNode field order](https://specs.ipfs.tech/unixfs/#dag-pb-node): `links-first` (canonical) or `data-first` (opt-in, see [IPIP-0550](https://specs.ipfs.tech/ipips/ipip-0550/)).
 
 ### Balanced DAG layout variants
 
@@ -214,6 +215,7 @@ Based on the research above, we define **`unixfs-v1-2025`** as an opinionated pr
 | Symlinks                      | preserved            |
 | Mode (permissions)            | excluded (opt-in)    |
 | Mtime (modification time)     | excluded (opt-in)    |
+| PBNode field order            | links-first          |
 
 ### The `unixfs-v0-2015` legacy profile
 
@@ -238,6 +240,7 @@ Note: this profile is a best-effort approximation of historical behavior. It pro
 | Symlinks                      | preserved            |
 | Mode (permissions)            | excluded (opt-in)    |
 | Mtime (modification time)     | excluded (opt-in)    |
+| PBNode field order            | links-first          |
 
 ## Design rationale
 

src/ipips/ipip-0550.md

@@ -0,0 +1,242 @@
+---
+title: "IPIP-0550: PBNode field ordering"
+date: 2026-09-07
+ipip: ratified
+editors:
+  - name: Alex Potsides
+    github: achingbrain
+    url: https://achingbrain.net/
+    affiliation:
+        name: Shipyard
+        url: https://ipshipyard.com
+  - name: Marcin Rataj
+    github: lidel
+    affiliation:
+      name: Shipyard
+      url: https://ipshipyard.com/
+relatedIssues:
+  - https://github.com/ipfs/specs/issues/533
+thanks:
+  - name: Rod Vagg
+    github: rvagg
+  - name: Volker Mische
+    github: vmx
+order: 550
+tags: ['ipips']
+---
+
+## Summary
+
+Add an opt-in `Data`-first field ordering for `PBNode` protobuf messages, so
+that streaming readers can process UnixFS metadata before links, and
+formalize that readers accept both orderings. No profile enables it: all
+profiles keep the canonical `Links`-first ordering and their CIDs, and
+writing `Data`-first requires an explicit opt-in setting.
+
+## Motivation
+
+Regular UnixFS and HAMT-sharded directories are encoded as `PBNode` protobuf
+messages.
+
+HAMT-sharded directory entries have the characteristic of prefixing the name of
+each entry with a number of characters drawn from the hash of the directory
+entry name.
+
+When multiple entries land in the same bucket, that bucket is replaced by a
+sub-shard with its own CID/block that contains those entries.
+
+The settings used to derive the prefix characters are stored in the `Data`
+field of the `PBNode` protobuf message.
+
+This means that all `Link` messages must be read from the `PBNode` message
+before we can read the `hashType` (a multihash code) and `fanout` values that
+let us calculate the prefix length for a given directory entry.
+
+When the reader is attempting to traverse to a single entry deep in the shard,
+they are forced to read all entries for the current sub-shard before they can
+move deeper within the shard, which leads to inefficient traversals.
+
+Independent of any efficiency gains, both orderings already exist in the
+wild: the DAG-PB codec specification requires decoders to accept either
+order "as IPFS data exists in both forms". Implementations comply, but by
+accident. Neither the UnixFS specification nor any test suite said so, and
+tolerance that is unspecified and untested is how interoperability decays:
+the code path works until the day it silently does not, and nothing catches
+it.
+
+The primary goal of this IPIP is the read side: every implementation can
+read UnixFS DAGs in either field order, and fixtures and conformance tests
+prove it. Producing the non-canonical `Data`-first order is optional and
+opt-in, but keeping a conforming writer around matters too: it is what keeps
+the tolerant read path exercised, so reading and writing such DAGs both stay
+interoperable instead of becoming an untested promise.
+
+This is the
+[robustness principle](https://specs.ipfs.tech/architecture/principles/#robustness)
+applied: "Be strict about the outcomes, be tolerant about the methods". The
+outcome is strict: the same logical UnixFS DAG, pinned by byte-exact
+fixtures and CIDs. The method, the field order on the wire, is tolerated in
+both forms, and, per the same principle's warning about silent tolerance,
+kept honest with tests rather than left to accident.
+
+## Detailed design
+
+If content authors are allowed to write the `Data` field first, readers can
+process `PBNode` messages with a streaming parser: the UnixFS metadata (for
+HAMT shards, the `hashType` and `fanout` parameters) arrives before the links,
+so a reader looking for a specific entry can stop reading links as soon as it
+finds the one it needs.
+
+### The `PBNode field order` parameter
+
+Writing the `Data` field first changes the CID generated for a piece of
+content, so this is an opt-in change.
+
+We add a new parameter to the set defined in
+[IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/):
+
+| Parameter          | Values                                  |
+| ------------------ | --------------------------------------- |
+| PBNode field order | `links-first` (default) or `data-first` |
+
+When a profile does not define `PBNode field order`, it is `links-first`.
+This IPIP amends the IPIP-0499 profile tables so `unixfs-v0-2015` and
+`unixfs-v1-2025` state `links-first` explicitly; they continue to produce the
+same CIDs as today.
+
+No profile sets `data-first` (see [Alternatives](#alternatives) for why).
+Implementations MAY expose an explicit low-level opt-in setting for writers
+that need it; enabling it changes the CID of every written node.
+
+### Changes to existing specifications
+
+This IPIP amends the [UnixFS specification](https://specs.ipfs.tech/unixfs/):
+
+- documents both `PBNode` field orderings: `Links`-first as the canonical
+  form, `Data`-first as the explicit opt-in
+- adds ordering requirements: readers SHOULD accept both orderings, and
+  implementations that interoperate with content on the public IPFS Mainnet
+  MUST accept both when reading; writers SHOULD support both orderings and
+  SHOULD write `Links`-first unless the user explicitly opted into
+  `Data`-first; specialized implementations MAY support a single ordering
+- adds a [Profiles](https://specs.ipfs.tech/unixfs/#profiles) section: a
+  registry of profile names that implementations SHOULD use in configuration,
+  flags, and test suites
+- adds test vectors covering both orderings
+
+It also amends the profile tables in
+[IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/) with an explicit
+`PBNode field order: links-first` row, so the preexisting profiles are
+protected from unintentional change.
+
+## Design rationale
+
+Traversing HAMT shards is more expensive than it needs to be, which
+disproportionately affects resource-constrained environments and inefficient
+runtimes.
+
+Working code: opt-in `data-first` writing merged in boxo
+([ipfs/boxo#1212](https://github.com/ipfs/boxo/pull/1212)) and Kubo
+([ipfs/kubo#11439](https://github.com/ipfs/kubo/pull/11439)), and shipped in
+[@ipld/dag-pb v4.2.0](https://github.com/ipld/js-dag-pb/releases/tag/v4.2.0)
+([ipld/js-dag-pb#111](https://github.com/ipld/js-dag-pb/pull/111));
+cross-ordering read tests and CAR fixtures shipped in
+[gateway-conformance v0.14.1](https://github.com/ipfs/gateway-conformance/releases/tag/v0.14.1).
+
+### User benefit
+
+Traversing HAMT shards will become faster in resource-constrained environments
+and inefficient runtimes.
+
+Existing content and workflows are unaffected: the canonical ordering remains
+the default everywhere, and an explicit opt-in is available for new
+developments that want the more efficient streaming reads.
+
+### Compatibility
+
+`PBNode` wire ordering is governed by the
+[DAG-PB codec specification](https://ipld.io/specs/codecs/dag-pb/spec/#protobuf-strictness),
+which requires decoders to accept both field orders ("as IPFS data exists in
+both forms") and, since [ipld/ipld#383](https://github.com/ipld/ipld/pull/383),
+explicitly permits encoders to write either order. Deployed implementations
+(go-codec-dagpb, js-dag-pb) already read both orders, so `Data`-first blocks
+are readable by existing software, and the two specifications agree. Should
+they diverge again, the [UnixFS specification](https://specs.ipfs.tech/unixfs/)
+governs UnixFS data.
+
+Backward compatibility is preserved by keeping the new ordering opt-in. Writers
+emit it only when a user explicitly enables the setting; every profile and
+default keeps the canonical ordering, so already-published CIDs and the CID
+determinism guarantees of `unixfs-v1-2025` are unchanged.
+
+### Security
+
+Two valid encodings for the same data are possible.
+
+- Same content, two CIDs, which harms deduplication, and CID-based denylists
+  need to include both CIDs.
+- Round trips from blocks to data and back change CIDs whenever the writer's
+  ordering differs from the ordering of the original block, in either
+  direction: a canonical writer re-encoding a `data-first` block, or a
+  `data-first` writer re-encoding a canonical block.
+  - This can reveal history of writes in systems like MFS.
+- Streaming readers MUST NOT act on links, for example by fetching child
+  blocks, before the enclosing block has been fully received and its multihash
+  verified.
+
+### Alternatives
+
+- **Change the ordering unconditionally.** Every implementation would start
+  producing different CIDs for the same input, breaking backward compatibility
+  and the CID determinism that `unixfs-v1-2025`
+  ([IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/)) guarantees.
+- **Flip the canonical encode order in the DAG-PB codec specification**, as
+  first proposed in [ipld/ipld#383](https://github.com/ipld/ipld/pull/383).
+  This has the same effect as changing the ordering unconditionally: writers
+  following the updated codec specification would silently produce new CIDs
+  for existing content. That PR was reworked to permit both orders without
+  changing the default, and merged in that form.
+- **Reader-side optimization without a format change.** `Links` fields are
+  length-delimited, so a reader holding a complete block can skip them cheaply
+  and read `Data` at the tail. This helps whole-block parsing, but does not
+  help streaming parsers, which cannot skip ahead in data that has not arrived
+  yet.
+- **Ship a `unixfs-v1-YYYY` profile that enables `data-first`.** Considered
+  and dropped: a dated successor to `unixfs-v1-2025` reads as "the
+  recommended latest", inviting unintentional adoption and a de facto new
+  CIDv1 default, exactly the CID churn this IPIP avoids. Creating a new CID
+  profile should be a separate IPIP, and its review should take a long time:
+  a new dated profile effectively informs the new suggested default for
+  CIDv1, and this IPIP deliberately changes no defaults. The parameter
+  machinery stays, so such a future IPIP can define the profile if
+  multi-implementation demand for deterministic `data-first` CIDs appears.
+
+## Test fixtures
+
+The table below lists minimal single-block vectors for both orderings. To
+verify compliance, decode each block, confirm the CID, and confirm that
+re-encoding the decoded node with the stated ordering reproduces the block
+bytes. Readers MUST resolve `hello.txt` through all four directory roots.
+
+| Description | Ordering | CID | Block (base16 encoded) |
+| --- | --- | --- | --- |
+| `hello.txt`, file content "hello\n" | raw leaf | `bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am` | `68656c6c6f0a` |
+| `Directory` containing `hello.txt` | `Data`-first (opt-in) | `bafybeigqvyloizmfcdy6scaxnyltftzptaruqa3hnnplfzsbf4sqteiwlm` | `0a02080112330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e7478741806` |
+| `Directory` containing `hello.txt` | `Links`-first (canonical) | `bafybeigdcg7pksx2zk5336vrfsktjodlr4rbfz37qr3koc5xboxe5ekv24` | `12330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e74787418060a020801` |
+| `HAMTShard` containing `hello.txt` | `Data`-first (hand-crafted) | `bafybeicwgy2rlqmqqu3yy2tqvm2wbgdvy3snu4sbbv4wqpvpnoplpzxz74` | `0a250805121c80000000000000000000000000000000000000000000000000000000282230800212350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e7478741806` |
+| `HAMTShard` containing `hello.txt` | `Links`-first (hand-crafted) | `bafybeicjwkfslu7gwyywffvqgse5kiibojtktxcdqhgv7ldj5fjdacuceq` | `12350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e74787418060a250805121c800000000000000000000000000000000000000000000000000000002822308002` |
+
+The `HAMTShard` blocks are hand-crafted for parser testing; no profile shards a
+single-entry directory via import (sharding starts above the 256 KiB
+threshold). The entry name `hello.txt` hashed with murmur3-x64-64 (`hashType`
+`0x22`) yields `0xDF` as the first byte, selecting bucket 223 at `fanout` 256:
+the link name is `DF` + `hello.txt`, and bit 223 is set in the
+`decode(PBNode.Data).Data` bitfield.
+
+All five blocks ship as
+[`pbnode-field-orders.car` in gateway-conformance v0.14.1](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.14.1/fixtures/path_gateway_unixfs/pbnode-field-orders.car),
+whose conformance tests exercise both orderings.
+
+### Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).