From 51f16e37f664f829a8187ed1fe2c0be62f5d9973 Mon Sep 17 00:00:00 2001 From: achingbrain Date: Mon, 24 Aug 2026 14:17:46 +0100 Subject: [PATCH 01/14] IPIP-550: PBNode field ordering Having `Link` messages before the `Data` message in `PBNode` messages means for HAMT shards you have to read every `Link` before you can process any of them, since the hash type/fanout values used to calculate the prefix each directory entry has is stored in the `Data` field at the end of the message. This IPIP suggests allowing writers to encode the `Data` field first, this allows readers to use streaming parsers that can skip out on reading links if they have already reached the directory entry they are interested in. Further discussion can be found in https://github.com/ipfs/specs/issues/533 --- src/ipips/ipip-0550.md | 85 ++++++++++++++++++++++++++++++++++++++++++ src/unixfs.md | 25 ++++++++++--- 2 files changed, 104 insertions(+), 6 deletions(-) create mode 100644 src/ipips/ipip-0550.md diff --git a/src/ipips/ipip-0550.md b/src/ipips/ipip-0550.md new file mode 100644 index 00000000..32130177 --- /dev/null +++ b/src/ipips/ipip-0550.md @@ -0,0 +1,85 @@ +--- +title: "IPIP-0550: PBNode field ordering" +date: 2026-08-24 +ipip: proposal +editors: + - name: Alex Potsides + github: achingbrain + url: https://achingbrain.net/ + affiliation: + name: Shipyard + url: https://ipshipyard.com +relatedIssues: + - https://github.com/ipfs/specs/issues/533 +order: 550 +tags: ['ipips'] +--- + +## Summary + +Encode `Data` field first in `PBNode` protobuf messages + +## Motivation + +Regular UnixFS and HAMT-sharded directories are encoded as `PBNode` protobuf +messages. + +HAMT-sharded directory entries have the characteristic of prefixing the name of +each entry with a number of characters drawn from the hash of the directory +entry name. + +Where hashes collide, a new sub-shard is created with it's own CID/block that +contains a sub-portion of the shard. + +The settings used to derive the prefix characters is stored in the `Data` field +of the `PBNode` protobuf message. + +This means that all `Link` messages must be read from the `PBNode` message +before we can read the hash algorithm name and fanout values that let us +calculate the prefix length for a given directory entry. + +When the reader is attempting to traverse to a single entry deep in the shard, +they are forced to read all entries for the current sub-shard before they can +move deeper within the shard, which leads to inefficient traversals. + +## Detailed design + +If we allow content authors to write the `Data` field first, readers can apply +a more efficient streaming parser for protobuf messages, since they will no +longer need to read all of the `Link` messages before they can process any of +them. + +## Design rationale + +Traversing HAMT shards is more expensive than it needs to be, which +disproportionately affects resource-constrained environments and inefficient +runtimes. + +### User benefit + +Traversing HAMT shards will become faster in resource-constrained environments +and inefficient runtimes. + +### Compatibility + +Protobuf has no requirement to write fields in any particular order, including +not in the numerical order of the field IDs, so this is a backwards compatible +change, unless custom protobuf parsers are used that expect fields defined in a +certain order. The UnixFS spec does not disallow this so any parsers enforcing +ordering on read may be considered buggy. + +### Security + +N/a + +### Alternatives + +N/a + +## Test fixtures + +N/a + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). diff --git a/src/unixfs.md b/src/unixfs.md index b9635376..b107990b 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -111,6 +111,19 @@ More complex nodes use the `dag-pb` (`0x70`) encoding. These nodes require two s decoding. The first step is to decode the outer container of the block. This is encoded using the [`dag-pb`][ipld-dag-pb] specification, which uses [Protocol Buffers][protobuf] and can be summarized as follows: +:::warning +In a earlier version of this spec, the `Data` field of the `PBNode` was ordered +after the repeated `Links` field. This lead to inefficiencies processing HAMT +shards as all links for a node must be read before the hash type and fanout +values could be read, which in turns means that if the ready is looking for a +specific path within the shard, they cannot abort reading links early. + +To support legacy data, implementations MUST be able to read and write `PBNode` +messages in the legacy format as well as the current format described below. + +Field IDs were the same in the legacy format. +::: + ```protobuf message PBLink { // Binary representation of CID (https://github.com/multiformats/cid) of the target object. @@ -127,11 +140,11 @@ message PBLink { } message PBNode { - // refs to other objects - repeated PBLink Links = 2; - // opaque user data bytes Data = 1; + + // refs to other objects + repeated PBLink Links = 2; } ``` @@ -368,12 +381,12 @@ The HAMT directory is configured through the UnixFS metadata in `PBNode.Data`: - `decode(PBNode.Data).fanout` is REQUIRED for HAMTShard nodes (though marked optional in the protobuf schema). The value MUST be a power of two, a multiple of 8 (for byte-aligned bitfields), and at most 1024. - + This determines the number of possible bucket indices (permutations) at each level of the trie. For example, fanout=256 provides 256 possible buckets (0x00 to 0xFF), requiring 8 bits from the hash. The hex prefix length is `log2(fanout)/4` characters (since each hex character represents 4 bits). The same fanout value is used throughout all levels of a single HAMT structure - + :::note Implementations that onboard user data to create new HAMTDirectory structures are free to choose a `fanout` value or allow users to configure it based on their use case: - **256**: Balanced tree depth and node size, suitable for most use cases @@ -382,7 +395,7 @@ The HAMT directory is configured through the UnixFS metadata in `PBNode.Data`: - Trade-offs: Larger blocks mean higher latency on cold cache reads and more data rewritten when modifying directories (each change affects a larger block) ::: - + :::warning Implementations MUST limit the `fanout` parameter to a maximum of 1024 to prevent denial-of-service attacks. Excessively large fanout values can cause memory exhaustion From 94edd62d3fd9447ce0ae946f716ffeb177fbf8ad Mon Sep 17 00:00:00 2001 From: achingbrain Date: Tue, 25 Aug 2026 09:55:31 +0100 Subject: [PATCH 02/14] chore: make writing legacy format optional --- src/unixfs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/unixfs.md b/src/unixfs.md index b107990b..efee25c8 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -118,7 +118,7 @@ shards as all links for a node must be read before the hash type and fanout values could be read, which in turns means that if the ready is looking for a specific path within the shard, they cannot abort reading links early. -To support legacy data, implementations MUST be able to read and write `PBNode` +To support legacy data and reproducible CIDs, implementations MAY write `PBNode` messages in the legacy format as well as the current format described below. Field IDs were the same in the legacy format. From d36b60821e7f5de57d991518865e4b7e6ecd9bf1 Mon Sep 17 00:00:00 2001 From: achingbrain Date: Tue, 25 Aug 2026 10:06:51 +0100 Subject: [PATCH 03/14] chore: add security considerations --- src/ipips/ipip-0550.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/ipips/ipip-0550.md b/src/ipips/ipip-0550.md index 32130177..7373a5b0 100644 --- a/src/ipips/ipip-0550.md +++ b/src/ipips/ipip-0550.md @@ -70,7 +70,13 @@ ordering on read may be considered buggy. ### Security -N/a +Two valid encodings for the same data are possible. + +- Same content, two CIDs which harms deduplication and CID-based denylists need to include both CIDs +- Round trips from blocks to data and back change CIDs if the writer does not use the legacy format + - This can reveal history of writes in systems like MFS + +Note that all of the above also applied to the introduction of metadata in UnixFS v1.5 which was deemed acceptable. ### Alternatives From 213b3268eb84cf05873b57f5f7568fd937f5ecd1 Mon Sep 17 00:00:00 2001 From: achingbrain Date: Tue, 25 Aug 2026 16:22:03 +0100 Subject: [PATCH 04/14] chore: add profile --- src/ipips/ipip-0550.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/src/ipips/ipip-0550.md b/src/ipips/ipip-0550.md index 7373a5b0..cebc0f8c 100644 --- a/src/ipips/ipip-0550.md +++ b/src/ipips/ipip-0550.md @@ -49,6 +49,22 @@ a more efficient streaming parser for protobuf messages, since they will no longer need to read all of the `Link` messages before they can process any of them. +### The `unixfs-v1-2026` profile + +Writing the `Data` field first will change the CID generated for a piece of +content, so this needs to be an opt-in change. + +We introduce a new **named configuration profiles** similar those added in +[IPIP-499](./ipip-0499.md). + +It inherits all settings from `unixfs-v1-2025` and adds a new parameter: + +| Parameter | `unixfs-v1-2026` | +| ----------------------------- | -------------------- | +| Write PBNode Data field first | true (opt-in) | + +If this parameter is not included in a profile it is assumed to be `false`. + ## Design rationale Traversing HAMT shards is more expensive than it needs to be, which From 546204bee1ccc3cf0bfe6838d05c49756b4caa9b Mon Sep 17 00:00:00 2001 From: achingbrain Date: Thu, 27 Aug 2026 09:34:31 +0100 Subject: [PATCH 05/14] chore: add fixtures --- src/ipips/ipip-0550.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/ipips/ipip-0550.md b/src/ipips/ipip-0550.md index cebc0f8c..ef88394d 100644 --- a/src/ipips/ipip-0550.md +++ b/src/ipips/ipip-0550.md @@ -89,18 +89,20 @@ ordering on read may be considered buggy. Two valid encodings for the same data are possible. - Same content, two CIDs which harms deduplication and CID-based denylists need to include both CIDs -- Round trips from blocks to data and back change CIDs if the writer does not use the legacy format +- Round trips from blocks to data and back change CIDs if the writer uses the legacy format - This can reveal history of writes in systems like MFS -Note that all of the above also applied to the introduction of metadata in UnixFS v1.5 which was deemed acceptable. - ### Alternatives N/a ## Test fixtures -N/a +| File | CID (unixfs-v1-2025 profile) | Block (base16 encoded) | +|---|---|---| +| hello.txt, basic text file | `bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am` | `68656c6c6f0a` | +| UnixFS directory containing `hello.txt`, with Data field before Links | `bafybeigqvyloizmfcdy6scaxnyltftzptaruqa3hnnplfzsbf4sqteiwlm` | `0a02080112330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e7478741806` | +| HAMT Shard containing `hello.txt`, with Data field before Links | `bafybeicwgy2rlqmqqu3yy2tqvm2wbgdvy3snu4sbbv4wqpvpnoplpzxz74` | `0a250805121c80000000000000000000000000000000000000000000000000000000282230800212350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e7478741806` | ### Copyright From 4fc886e2f3f4f4692a4d673ded60ea035d2e525e Mon Sep 17 00:00:00 2001 From: achingbrain Date: Thu, 27 Aug 2026 09:38:36 +0100 Subject: [PATCH 06/14] chore: add legacy fixtures --- src/ipips/ipip-0550.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/ipips/ipip-0550.md b/src/ipips/ipip-0550.md index ef88394d..ce8ec7eb 100644 --- a/src/ipips/ipip-0550.md +++ b/src/ipips/ipip-0550.md @@ -102,7 +102,9 @@ N/a |---|---|---| | hello.txt, basic text file | `bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am` | `68656c6c6f0a` | | UnixFS directory containing `hello.txt`, with Data field before Links | `bafybeigqvyloizmfcdy6scaxnyltftzptaruqa3hnnplfzsbf4sqteiwlm` | `0a02080112330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e7478741806` | +| UnixFS directory containing `hello.txt`, legacy format with Links before Data field | `bafybeigdcg7pksx2zk5336vrfsktjodlr4rbfz37qr3koc5xboxe5ekv24` | `12330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e74787418060a020801` | | HAMT Shard containing `hello.txt`, with Data field before Links | `bafybeicwgy2rlqmqqu3yy2tqvm2wbgdvy3snu4sbbv4wqpvpnoplpzxz74` | `0a250805121c80000000000000000000000000000000000000000000000000000000282230800212350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e7478741806` | +| HAMT Shard containing `hello.txt`, legacy format with Links before Data field | `bafybeicjwkfslu7gwyywffvqgse5kiibojtktxcdqhgv7ldj5fjdacuceq` | `12350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e74787418060a250805121c800000000000000000000000000000000000000000000000000000002822308002` | ### Copyright From 17444efd86d0043500d35cba4eda25a7be584314 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 27 Aug 2026 21:59:17 +0200 Subject: [PATCH 07/14] docs: opt-in ordering and profiles registry Make the Data-first ordering opt-in per profile and align the text with DAG-PB codec behavior. - unixfs.md: Profiles registry section with names implementations SHOULD use; both PBNode orderings with reader/writer rules and UnixFS precedence over the legacy DAG-PB codec spec; data-first test vectors in the appendix - ipip-0550: accurate motivation and compatibility, PBNode field order profile parameter (links-first default), two-way security notes, alternatives, fixtures relabeled and linked to tagged gateway-conformance v0.14.1 CARs, working code links --- src/ipips/ipip-0550.md | 156 +++++++++++++++++++++++++++++++---------- src/unixfs.md | 99 ++++++++++++++++++++++---- 2 files changed, 204 insertions(+), 51 deletions(-) diff --git a/src/ipips/ipip-0550.md b/src/ipips/ipip-0550.md index ce8ec7eb..5360ec60 100644 --- a/src/ipips/ipip-0550.md +++ b/src/ipips/ipip-0550.md @@ -1,6 +1,6 @@ --- title: "IPIP-0550: PBNode field ordering" -date: 2026-08-24 +date: 2026-08-27 ipip: proposal editors: - name: Alex Potsides @@ -17,7 +17,10 @@ tags: ['ipips'] ## Summary -Encode `Data` field first in `PBNode` protobuf messages +Add an opt-in `Data`-first field ordering for `PBNode` protobuf messages, +enabled by a new `unixfs-v1-2026` profile, so that streaming readers can +process UnixFS metadata before links. All existing profiles keep the legacy +`Links`-first ordering, so already-published CIDs are unaffected. ## Motivation @@ -28,15 +31,15 @@ HAMT-sharded directory entries have the characteristic of prefixing the name of each entry with a number of characters drawn from the hash of the directory entry name. -Where hashes collide, a new sub-shard is created with it's own CID/block that -contains a sub-portion of the shard. +When multiple entries land in the same bucket, that bucket is replaced by a +sub-shard with its own CID/block that contains those entries. -The settings used to derive the prefix characters is stored in the `Data` field -of the `PBNode` protobuf message. +The settings used to derive the prefix characters are stored in the `Data` +field of the `PBNode` protobuf message. This means that all `Link` messages must be read from the `PBNode` message -before we can read the hash algorithm name and fanout values that let us -calculate the prefix length for a given directory entry. +before we can read the `hashType` (a multihash code) and `fanout` values that +let us calculate the prefix length for a given directory entry. When the reader is attempting to traverse to a single entry deep in the shard, they are forced to read all entries for the current sub-shard before they can @@ -44,26 +47,47 @@ move deeper within the shard, which leads to inefficient traversals. ## Detailed design -If we allow content authors to write the `Data` field first, readers can apply -a more efficient streaming parser for protobuf messages, since they will no -longer need to read all of the `Link` messages before they can process any of -them. +If content authors are allowed to write the `Data` field first, readers can +process `PBNode` messages with a streaming parser: the UnixFS metadata (for +HAMT shards, the `hashType` and `fanout` parameters) arrives before the links, +so a reader looking for a specific entry can stop reading links as soon as it +finds the one it needs. ### The `unixfs-v1-2026` profile -Writing the `Data` field first will change the CID generated for a piece of -content, so this needs to be an opt-in change. +Writing the `Data` field first changes the CID generated for a piece of +content, so this is an opt-in change. -We introduce a new **named configuration profiles** similar those added in -[IPIP-499](./ipip-0499.md). +We introduce a new **named configuration profile** similar to those added in +[IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/). -It inherits all settings from `unixfs-v1-2025` and adds a new parameter: +It inherits all settings from `unixfs-v1-2025` (as ratified in IPIP-0499) and +adds a new parameter: -| Parameter | `unixfs-v1-2026` | -| ----------------------------- | -------------------- | -| Write PBNode Data field first | true (opt-in) | +| Parameter | `unixfs-v1-2026` | +| ------------------ | ---------------- | +| PBNode field order | `data-first` | -If this parameter is not included in a profile it is assumed to be `false`. +When a profile does not define this parameter, it is `links-first`. All +profiles defined before this IPIP, including `unixfs-v0-2015` and +`unixfs-v1-2025`, keep the legacy `Links`-first ordering and continue to +produce the same CIDs as today. + +### Changes to the UnixFS specification + +This IPIP amends the [UnixFS specification](https://specs.ipfs.tech/unixfs/): + +- documents both `PBNode` field orderings: `Data`-first written under + `unixfs-v1-2026`, `Links`-first written under the earlier profiles +- adds ordering requirements: readers SHOULD accept both orderings, and + implementations that interoperate with content on the public IPFS Mainnet + MUST accept both when reading; writers SHOULD support both orderings and + write the one mandated by the selected profile; specialized implementations + MAY support a single ordering +- adds a [Profiles](https://specs.ipfs.tech/unixfs/#profiles) section: a + registry of profile names that implementations SHOULD use in configuration, + flags, and test suites +- adds test vectors covering both orderings ## Design rationale @@ -71,40 +95,98 @@ Traversing HAMT shards is more expensive than it needs to be, which disproportionately affects resource-constrained environments and inefficient runtimes. +Working code: opt-in `data-first` writing in boxo ([ipfs/boxo#1212](https://github.com/ipfs/boxo/pull/1212)) +and Kubo ([ipfs/kubo#11439](https://github.com/ipfs/kubo/pull/11439)); +cross-ordering read tests and CAR fixtures shipped in +[gateway-conformance v0.14.1](https://github.com/ipfs/gateway-conformance/releases/tag/v0.14.1). + ### User benefit Traversing HAMT shards will become faster in resource-constrained environments and inefficient runtimes. +Existing content and workflows are unaffected: the legacy ordering remains +fully supported, and `unixfs-v1-2026` is available for new developments that +want the more efficient streaming reads. + ### Compatibility -Protobuf has no requirement to write fields in any particular order, including -not in the numerical order of the field IDs, so this is a backwards compatible -change, unless custom protobuf parsers are used that expect fields defined in a -certain order. The UnixFS spec does not disallow this so any parsers enforcing -ordering on read may be considered buggy. +`PBNode` wire ordering is governed by the +[DAG-PB codec specification](https://ipld.io/specs/codecs/dag-pb/spec/#protobuf-strictness), +which requires decoders to accept both field orders ("as IPFS data exists in +both forms") while mandating `Links`-first on encode. Deployed implementations +(go-codec-dagpb, js-dag-pb) already read both orders, so `Data`-first blocks +are readable by existing software. + +On the write side, `Data`-first deviates from the canonical encode order of the +historical DAG-PB codec specification. For UnixFS data, the +[UnixFS specification](https://specs.ipfs.tech/unixfs/) takes precedence over +the historical DAG-PB codec specification: blocks produced under the +`unixfs-v1-2026` profile are valid UnixFS. + +Backward compatibility is preserved by keeping the new ordering opt-in. Writers +emit it only when a user explicitly selects `unixfs-v1-2026`; every existing +profile and default keeps the legacy ordering, so already-published CIDs and +the CID determinism guarantees of `unixfs-v1-2025` are unchanged. ### Security Two valid encodings for the same data are possible. -- Same content, two CIDs which harms deduplication and CID-based denylists need to include both CIDs -- Round trips from blocks to data and back change CIDs if the writer uses the legacy format - - This can reveal history of writes in systems like MFS +- Same content, two CIDs, which harms deduplication, and CID-based denylists + need to include both CIDs. +- Round trips from blocks to data and back change CIDs whenever the writer's + ordering differs from the ordering of the original block, in either + direction: a legacy writer re-encoding a `data-first` block, or a + `unixfs-v1-2026` writer re-encoding a legacy block. + - This can reveal history of writes in systems like MFS. +- Streaming readers MUST NOT act on links, for example by fetching child + blocks, before the enclosing block has been fully received and its multihash + verified. ### Alternatives -N/a +- **Change the ordering unconditionally.** Every implementation would start + producing different CIDs for the same input, breaking backward compatibility + and the CID determinism that `unixfs-v1-2025` + ([IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/)) guarantees. +- **Change the canonical encode order in the DAG-PB codec specification** + ([ipld/ipld#383](https://github.com/ipld/ipld/pull/383)). This has the same + effect as changing the ordering unconditionally: writers following the + updated codec specification would silently produce new CIDs for existing + content. Ordering must remain an opt-in, per-profile choice at the UnixFS + layer. +- **Reader-side optimization without a format change.** `Links` fields are + length-delimited, so a reader holding a complete block can skip them cheaply + and read `Data` at the tail. This helps whole-block parsing, but does not + help streaming parsers, which cannot skip ahead in data that has not arrived + yet. ## Test fixtures -| File | CID (unixfs-v1-2025 profile) | Block (base16 encoded) | -|---|---|---| -| hello.txt, basic text file | `bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am` | `68656c6c6f0a` | -| UnixFS directory containing `hello.txt`, with Data field before Links | `bafybeigqvyloizmfcdy6scaxnyltftzptaruqa3hnnplfzsbf4sqteiwlm` | `0a02080112330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e7478741806` | -| UnixFS directory containing `hello.txt`, legacy format with Links before Data field | `bafybeigdcg7pksx2zk5336vrfsktjodlr4rbfz37qr3koc5xboxe5ekv24` | `12330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e74787418060a020801` | -| HAMT Shard containing `hello.txt`, with Data field before Links | `bafybeicwgy2rlqmqqu3yy2tqvm2wbgdvy3snu4sbbv4wqpvpnoplpzxz74` | `0a250805121c80000000000000000000000000000000000000000000000000000000282230800212350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e7478741806` | -| HAMT Shard containing `hello.txt`, legacy format with Links before Data field | `bafybeicjwkfslu7gwyywffvqgse5kiibojtktxcdqhgv7ldj5fjdacuceq` | `12350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e74787418060a250805121c800000000000000000000000000000000000000000000000000000002822308002` | +The table below lists minimal single-block vectors for both orderings. To +verify compliance, decode each block, confirm the CID, and confirm that +re-encoding the decoded node with the stated ordering reproduces the block +bytes. Readers MUST resolve `hello.txt` through all four directory roots. + +| Description | Ordering (profile) | CID | Block (base16 encoded) | +| --- | --- | --- | --- | +| `hello.txt`, file content "hello\n" | raw leaf (any profile) | `bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am` | `68656c6c6f0a` | +| `Directory` containing `hello.txt` | `Data`-first (`unixfs-v1-2026`) | `bafybeigqvyloizmfcdy6scaxnyltftzptaruqa3hnnplfzsbf4sqteiwlm` | `0a02080112330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e7478741806` | +| `Directory` containing `hello.txt` | `Links`-first (`unixfs-v1-2025`) | `bafybeigdcg7pksx2zk5336vrfsktjodlr4rbfz37qr3koc5xboxe5ekv24` | `12330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e74787418060a020801` | +| `HAMTShard` containing `hello.txt` | `Data`-first (hand-crafted) | `bafybeicwgy2rlqmqqu3yy2tqvm2wbgdvy3snu4sbbv4wqpvpnoplpzxz74` | `0a250805121c80000000000000000000000000000000000000000000000000000000282230800212350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e7478741806` | +| `HAMTShard` containing `hello.txt` | `Links`-first (hand-crafted) | `bafybeicjwkfslu7gwyywffvqgse5kiibojtktxcdqhgv7ldj5fjdacuceq` | `12350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e74787418060a250805121c800000000000000000000000000000000000000000000000000000002822308002` | + +The `HAMTShard` blocks are hand-crafted for parser testing; no profile shards a +single-entry directory via import (sharding starts above the 256 KiB +threshold). The entry name `hello.txt` hashed with murmur3-x64-64 (`hashType` +`0x22`) yields `0xDF` as the first byte, selecting bucket 223 at `fanout` 256: +the link name is `DF` + `hello.txt`, and bit 223 is set in the +`decode(PBNode.Data).Data` bitfield. + +All five blocks ship as +[`pbnode-field-orders.car` in gateway-conformance v0.14.1](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.14.1/fixtures/path_gateway_unixfs/pbnode-field-orders.car), +whose conformance tests exercise both orderings. ### Copyright diff --git a/src/unixfs.md b/src/unixfs.md index efee25c8..2589b47a 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -3,7 +3,7 @@ title: UnixFS description: > UnixFS is a Protocol Buffers-based format for describing files and directories as dag-pb DAGs and raw blocks in IPFS. -date: 2026-03-05 +date: 2026-08-27 maturity: draft editors: - name: Marcin Rataj @@ -111,19 +111,6 @@ More complex nodes use the `dag-pb` (`0x70`) encoding. These nodes require two s decoding. The first step is to decode the outer container of the block. This is encoded using the [`dag-pb`][ipld-dag-pb] specification, which uses [Protocol Buffers][protobuf] and can be summarized as follows: -:::warning -In a earlier version of this spec, the `Data` field of the `PBNode` was ordered -after the repeated `Links` field. This lead to inefficiencies processing HAMT -shards as all links for a node must be read before the hash type and fanout -values could be read, which in turns means that if the ready is looking for a -specific path within the shard, they cannot abort reading links early. - -To support legacy data and reproducible CIDs, implementations MAY write `PBNode` -messages in the legacy format as well as the current format described below. - -Field IDs were the same in the legacy format. -::: - ```protobuf message PBLink { // Binary representation of CID (https://github.com/multiformats/cid) of the target object. @@ -138,7 +125,34 @@ message PBLink { // cumulative size of target object uint64 Tsize = 3; } +``` + +The `PBNode` message holds two fields, `Data` (field number 1) and `Links` +(field number 2). Two wire orderings of these fields exist, and both decode to +the same logical node. A conforming writer emits the fields in the declaration +order shown in the chosen variant below. + +:::warning +The two orderings produce different bytes, and therefore different CIDs, for +the same logical node. Which ordering a writer emits is controlled by the +selected [profile](#profiles): + +- `Data`-first is written under the `unixfs-v1-2026` and later profiles. +- `Links`-first is the legacy ordering written under the `unixfs-v0-2015` and + `unixfs-v1-2025` profiles, and the canonical field order of the historical + [DAG-PB][ipld-dag-pb] codec specification. For UnixFS data, this document + takes precedence over the historical DAG-PB codec specification. + +Readers SHOULD accept both orderings. Writers SHOULD support both orderings +and write the ordering mandated by the selected profile. Specialized +implementations MAY support a single ordering, for example a streaming-oriented +producer that only emits `Data`-first. Implementations that interoperate with +content on the public IPFS Mainnet MUST accept both orderings when reading. +::: +`Data`-first ordering, written under the `unixfs-v1-2026` and later profiles: + +```protobuf message PBNode { // opaque user data bytes Data = 1; @@ -148,6 +162,24 @@ message PBNode { } ``` +`Links`-first ordering (legacy), written under the `unixfs-v0-2015` and +`unixfs-v1-2025` profiles: + +```protobuf +message PBNode { + // refs to other objects + repeated PBLink Links = 2; + + // opaque user data + bytes Data = 1; +} +``` + +The `Data`-first ordering lets a streaming reader process the UnixFS metadata +in `Data` (for example, HAMTShard `hashType` and `fanout`) before the links, +and stop reading links early once it finds the entry it is looking for. The +legacy ordering keeps the CIDs of already-published content stable. + After decoding the node, we obtain a `PBNode`. This `PBNode` contains a field `Data` that contains the bytes that require the second decoding. This will also be a protobuf message specified in the UnixFSV1 format: @@ -726,6 +758,26 @@ The following names SHOULD NOT be used in UnixFS directories: terminations in some systems, such as C-compatible systems. Many unix file systems do not accept this character in path components. +# Profiles + +DAG construction parameters such as chunk size, DAG width, HAMT fanout and +threshold, and `PBNode` field ordering all affect the resulting CID, so the +same input can produce different CIDs across implementations. A :dfn[Profile] +is a named, complete set of these parameters: two implementations importing +the same input under the same profile produce the same CID. + +The following profiles are defined: + +| Profile | Defined in | Description | +| --- | --- | --- | +| `unixfs-v0-2015` | [IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/) | Legacy CIDv0 parameters matching Kubo defaults through v0.39. For reproducing historical CIDv0 references. | +| `unixfs-v1-2025` | [IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/) | Deterministic CIDv1 parameters with modern settings. Writes the legacy `Links`-first `PBNode` ordering. | +| `unixfs-v1-2026` | [IPIP-0550](https://specs.ipfs.tech/ipips/ipip-0550/) | Same as `unixfs-v1-2025`, plus the `Data`-first `PBNode` ordering for more efficient streaming reads. | + +Implementations SHOULD use these exact profile names when exposing profile +selection in configuration, command-line flags, documentation, and test +suites, so that a profile name means the same thing in every tool. + # Appendix: Test Vectors :::warning @@ -972,6 +1024,25 @@ Test vectors for UnixFS directory structures, progressing from simple flat direc - Link Names in HAMT have 2-character hex prefix (hash buckets) - Can retrieve any file by name through hash bucket calculation +### PBNode Field Ordering + +- Fixture: [`pbnode-field-orders.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.14.1/fixtures/path_gateway_unixfs/pbnode-field-orders.car) + - Type: [`dag-pb` Directory](#dag-pb-directory) and + [`dag-pb` HAMTDirectory](#dag-pb-hamtdirectory) in both `PBNode` field + orderings (see [`dag-pb` Node](#dag-pb-node) and [Profiles](#profiles)) + - CIDs: + - `bafybeigqvyloizmfcdy6scaxnyltftzptaruqa3hnnplfzsbf4sqteiwlm`: `Directory`, `Data`-first (`unixfs-v1-2026`) + - `bafybeigdcg7pksx2zk5336vrfsktjodlr4rbfz37qr3koc5xboxe5ekv24`: `Directory`, `Links`-first (`unixfs-v1-2025`) + - `bafybeicwgy2rlqmqqu3yy2tqvm2wbgdvy3snu4sbbv4wqpvpnoplpzxz74`: `HAMTShard`, `Data`-first (hand-crafted) + - `bafybeicjwkfslu7gwyywffvqgse5kiibojtktxcdqhgv7ldj5fjdacuceq`: `HAMTShard`, `Links`-first (hand-crafted) + - Contents: each root holds a single `hello.txt` ("hello\n") stored as a + `raw` leaf (`bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am`) + - Purpose: readers accept both `PBNode` field orderings + - Validation: + - `hello.txt` resolves through all four roots + - Byte-level vectors: fixtures table in + [IPIP-0550](https://specs.ipfs.tech/ipips/ipip-0550/) + ## Special Cases and Advanced Features Test vectors for special UnixFS features and edge cases. From b7072070636fdf9ad90313a49893c4012bd7c767 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 27 Aug 2026 22:11:01 +0200 Subject: [PATCH 08/14] docs: full unixfs-v1-2026 parameter table --- src/ipips/ipip-0550.md | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/src/ipips/ipip-0550.md b/src/ipips/ipip-0550.md index 5360ec60..61f2a6ed 100644 --- a/src/ipips/ipip-0550.md +++ b/src/ipips/ipip-0550.md @@ -62,13 +62,30 @@ We introduce a new **named configuration profile** similar to those added in [IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/). It inherits all settings from `unixfs-v1-2025` (as ratified in IPIP-0499) and -adds a new parameter: - -| Parameter | `unixfs-v1-2026` | -| ------------------ | ---------------- | -| PBNode field order | `data-first` | - -When a profile does not define this parameter, it is `links-first`. All +adds one parameter, `PBNode field order`. The complete profile: + +| Parameter | `unixfs-v1-2026` | +| ----------------------------- | -------------------- | +| CID version | CIDv1 | +| Hash function | sha2-256 | +| Chunking algorithm | fixed-size | +| Max chunk size | 1MiB | +| DAG layout | balanced | +| DAG width (children per node) | 1024 | +| HAMTDirectory fanout | 256 blocks | +| HAMTDirectory threshold | 256KiB (block-bytes) | +| HAMT switch comparison | > | +| Leaves | raw | +| Empty directories | included (opt-out) | +| Hidden entities | excluded (opt-in) | +| Symlinks | preserved | +| Mode (permissions) | excluded (opt-in) | +| Mtime (modification time) | excluded (opt-in) | +| PBNode field order | `data-first` | + +Only the last row is new; every other value matches `unixfs-v1-2025`. + +When a profile does not define `PBNode field order`, it is `links-first`. All profiles defined before this IPIP, including `unixfs-v0-2015` and `unixfs-v1-2025`, keep the legacy `Links`-first ordering and continue to produce the same CIDs as today. From d99d0ee9c07f118ed59b9f64f3f1c4f7cc15f0c9 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Wed, 2 Sep 2026 14:48:58 +0200 Subject: [PATCH 09/14] docs: drop 2026 profile, keep opt-in parameter No new profile: a dated successor to unixfs-v1-2025 reads as "the recommended latest" and invites unintentional adoption of a new de facto CIDv1 default. Data-first stays an explicit low-level opt-in. - ipip-0499: explicit "PBNode field order: links-first" row in both profile tables, protecting them from unintended change - ipip-0550: defines the parameter instead of a profile and records the dropped profile under Alternatives - unixfs.md: links-first canonical, data-first opt-in; profile registry keeps 2015/2025 only --- src/ipips/ipip-0499.md | 2 + src/ipips/ipip-0550.md | 116 ++++++++++++++++++++--------------------- src/unixfs.md | 40 +++++++------- 3 files changed, 79 insertions(+), 79 deletions(-) diff --git a/src/ipips/ipip-0499.md b/src/ipips/ipip-0499.md index eb94da2d..31540246 100644 --- a/src/ipips/ipip-0499.md +++ b/src/ipips/ipip-0499.md @@ -214,6 +214,7 @@ Based on the research above, we define **`unixfs-v1-2025`** as an opinionated pr | Symlinks | preserved | | Mode (permissions) | excluded (opt-in) | | Mtime (modification time) | excluded (opt-in) | +| PBNode field order | links-first | ### The `unixfs-v0-2015` legacy profile @@ -238,6 +239,7 @@ Note: this profile is a best-effort approximation of historical behavior. It pro | Symlinks | preserved | | Mode (permissions) | excluded (opt-in) | | Mtime (modification time) | excluded (opt-in) | +| PBNode field order | links-first | ## Design rationale diff --git a/src/ipips/ipip-0550.md b/src/ipips/ipip-0550.md index 61f2a6ed..ca4bcb56 100644 --- a/src/ipips/ipip-0550.md +++ b/src/ipips/ipip-0550.md @@ -1,6 +1,6 @@ --- title: "IPIP-0550: PBNode field ordering" -date: 2026-08-27 +date: 2026-09-01 ipip: proposal editors: - name: Alex Potsides @@ -17,10 +17,11 @@ tags: ['ipips'] ## Summary -Add an opt-in `Data`-first field ordering for `PBNode` protobuf messages, -enabled by a new `unixfs-v1-2026` profile, so that streaming readers can -process UnixFS metadata before links. All existing profiles keep the legacy -`Links`-first ordering, so already-published CIDs are unaffected. +Add an opt-in `Data`-first field ordering for `PBNode` protobuf messages, so +that streaming readers can process UnixFS metadata before links, and +formalize that readers accept both orderings. No profile enables it: all +profiles keep the canonical `Links`-first ordering and their CIDs, and +writing `Data`-first requires an explicit opt-in setting. ## Motivation @@ -53,59 +54,48 @@ HAMT shards, the `hashType` and `fanout` parameters) arrives before the links, so a reader looking for a specific entry can stop reading links as soon as it finds the one it needs. -### The `unixfs-v1-2026` profile +### The `PBNode field order` parameter Writing the `Data` field first changes the CID generated for a piece of content, so this is an opt-in change. -We introduce a new **named configuration profile** similar to those added in -[IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/). - -It inherits all settings from `unixfs-v1-2025` (as ratified in IPIP-0499) and -adds one parameter, `PBNode field order`. The complete profile: - -| Parameter | `unixfs-v1-2026` | -| ----------------------------- | -------------------- | -| CID version | CIDv1 | -| Hash function | sha2-256 | -| Chunking algorithm | fixed-size | -| Max chunk size | 1MiB | -| DAG layout | balanced | -| DAG width (children per node) | 1024 | -| HAMTDirectory fanout | 256 blocks | -| HAMTDirectory threshold | 256KiB (block-bytes) | -| HAMT switch comparison | > | -| Leaves | raw | -| Empty directories | included (opt-out) | -| Hidden entities | excluded (opt-in) | -| Symlinks | preserved | -| Mode (permissions) | excluded (opt-in) | -| Mtime (modification time) | excluded (opt-in) | -| PBNode field order | `data-first` | - -Only the last row is new; every other value matches `unixfs-v1-2025`. - -When a profile does not define `PBNode field order`, it is `links-first`. All -profiles defined before this IPIP, including `unixfs-v0-2015` and -`unixfs-v1-2025`, keep the legacy `Links`-first ordering and continue to -produce the same CIDs as today. - -### Changes to the UnixFS specification +We add a new parameter to the set defined in +[IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/): + +| Parameter | Values | +| ------------------ | --------------------------------------- | +| PBNode field order | `links-first` (default) or `data-first` | + +When a profile does not define `PBNode field order`, it is `links-first`. +This IPIP amends the IPIP-0499 profile tables so `unixfs-v0-2015` and +`unixfs-v1-2025` state `links-first` explicitly; they continue to produce the +same CIDs as today. + +No profile sets `data-first` (see [Alternatives](#alternatives) for why). +Implementations MAY expose an explicit low-level opt-in setting for writers +that need it; enabling it changes the CID of every written node. + +### Changes to existing specifications This IPIP amends the [UnixFS specification](https://specs.ipfs.tech/unixfs/): -- documents both `PBNode` field orderings: `Data`-first written under - `unixfs-v1-2026`, `Links`-first written under the earlier profiles +- documents both `PBNode` field orderings: `Links`-first as the canonical + form, `Data`-first as the explicit opt-in - adds ordering requirements: readers SHOULD accept both orderings, and implementations that interoperate with content on the public IPFS Mainnet MUST accept both when reading; writers SHOULD support both orderings and - write the one mandated by the selected profile; specialized implementations - MAY support a single ordering + SHOULD write `Links`-first unless the user explicitly opted into + `Data`-first; specialized implementations MAY support a single ordering - adds a [Profiles](https://specs.ipfs.tech/unixfs/#profiles) section: a registry of profile names that implementations SHOULD use in configuration, flags, and test suites - adds test vectors covering both orderings +It also amends the profile tables in +[IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/) with an explicit +`PBNode field order: links-first` row, so the preexisting profiles are +protected from unintentional change. + ## Design rationale Traversing HAMT shards is more expensive than it needs to be, which @@ -122,9 +112,9 @@ cross-ordering read tests and CAR fixtures shipped in Traversing HAMT shards will become faster in resource-constrained environments and inefficient runtimes. -Existing content and workflows are unaffected: the legacy ordering remains -fully supported, and `unixfs-v1-2026` is available for new developments that -want the more efficient streaming reads. +Existing content and workflows are unaffected: the canonical ordering remains +the default everywhere, and an explicit opt-in is available for new +developments that want the more efficient streaming reads. ### Compatibility @@ -138,13 +128,13 @@ are readable by existing software. On the write side, `Data`-first deviates from the canonical encode order of the historical DAG-PB codec specification. For UnixFS data, the [UnixFS specification](https://specs.ipfs.tech/unixfs/) takes precedence over -the historical DAG-PB codec specification: blocks produced under the -`unixfs-v1-2026` profile are valid UnixFS. +the historical DAG-PB codec specification: blocks produced with the opt-in +`data-first` setting are valid UnixFS. Backward compatibility is preserved by keeping the new ordering opt-in. Writers -emit it only when a user explicitly selects `unixfs-v1-2026`; every existing -profile and default keeps the legacy ordering, so already-published CIDs and -the CID determinism guarantees of `unixfs-v1-2025` are unchanged. +emit it only when a user explicitly enables the setting; every profile and +default keeps the canonical ordering, so already-published CIDs and the CID +determinism guarantees of `unixfs-v1-2025` are unchanged. ### Security @@ -154,8 +144,8 @@ Two valid encodings for the same data are possible. need to include both CIDs. - Round trips from blocks to data and back change CIDs whenever the writer's ordering differs from the ordering of the original block, in either - direction: a legacy writer re-encoding a `data-first` block, or a - `unixfs-v1-2026` writer re-encoding a legacy block. + direction: a canonical writer re-encoding a `data-first` block, or a + `data-first` writer re-encoding a canonical block. - This can reveal history of writes in systems like MFS. - Streaming readers MUST NOT act on links, for example by fetching child blocks, before the enclosing block has been fully received and its multihash @@ -171,13 +161,21 @@ Two valid encodings for the same data are possible. ([ipld/ipld#383](https://github.com/ipld/ipld/pull/383)). This has the same effect as changing the ordering unconditionally: writers following the updated codec specification would silently produce new CIDs for existing - content. Ordering must remain an opt-in, per-profile choice at the UnixFS - layer. + content. Ordering must remain an opt-in choice at the UnixFS layer. - **Reader-side optimization without a format change.** `Links` fields are length-delimited, so a reader holding a complete block can skip them cheaply and read `Data` at the tail. This helps whole-block parsing, but does not help streaming parsers, which cannot skip ahead in data that has not arrived yet. +- **Ship a `unixfs-v1-YYYY` profile that enables `data-first`.** Considered + and dropped: a dated successor to `unixfs-v1-2025` reads as "the + recommended latest", inviting unintentional adoption and a de facto new + CIDv1 default, exactly the CID churn this IPIP avoids. Creating a new CID + profile should be a separate IPIP, and its review should take a long time: + a new dated profile effectively informs the new suggested default for + CIDv1, and this IPIP deliberately changes no defaults. The parameter + machinery stays, so such a future IPIP can define the profile if + multi-implementation demand for deterministic `data-first` CIDs appears. ## Test fixtures @@ -186,11 +184,11 @@ verify compliance, decode each block, confirm the CID, and confirm that re-encoding the decoded node with the stated ordering reproduces the block bytes. Readers MUST resolve `hello.txt` through all four directory roots. -| Description | Ordering (profile) | CID | Block (base16 encoded) | +| Description | Ordering | CID | Block (base16 encoded) | | --- | --- | --- | --- | -| `hello.txt`, file content "hello\n" | raw leaf (any profile) | `bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am` | `68656c6c6f0a` | -| `Directory` containing `hello.txt` | `Data`-first (`unixfs-v1-2026`) | `bafybeigqvyloizmfcdy6scaxnyltftzptaruqa3hnnplfzsbf4sqteiwlm` | `0a02080112330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e7478741806` | -| `Directory` containing `hello.txt` | `Links`-first (`unixfs-v1-2025`) | `bafybeigdcg7pksx2zk5336vrfsktjodlr4rbfz37qr3koc5xboxe5ekv24` | `12330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e74787418060a020801` | +| `hello.txt`, file content "hello\n" | raw leaf | `bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am` | `68656c6c6f0a` | +| `Directory` containing `hello.txt` | `Data`-first (opt-in) | `bafybeigqvyloizmfcdy6scaxnyltftzptaruqa3hnnplfzsbf4sqteiwlm` | `0a02080112330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e7478741806` | +| `Directory` containing `hello.txt` | `Links`-first (canonical) | `bafybeigdcg7pksx2zk5336vrfsktjodlr4rbfz37qr3koc5xboxe5ekv24` | `12330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e74787418060a020801` | | `HAMTShard` containing `hello.txt` | `Data`-first (hand-crafted) | `bafybeicwgy2rlqmqqu3yy2tqvm2wbgdvy3snu4sbbv4wqpvpnoplpzxz74` | `0a250805121c80000000000000000000000000000000000000000000000000000000282230800212350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e7478741806` | | `HAMTShard` containing `hello.txt` | `Links`-first (hand-crafted) | `bafybeicjwkfslu7gwyywffvqgse5kiibojtktxcdqhgv7ldj5fjdacuceq` | `12350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e74787418060a250805121c800000000000000000000000000000000000000000000000000000002822308002` | diff --git a/src/unixfs.md b/src/unixfs.md index a2d8e57b..c5353ef9 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -3,7 +3,7 @@ title: UnixFS description: > UnixFS is a Protocol Buffers-based format for describing files and directories as dag-pb DAGs and raw blocks in IPFS. -date: 2026-08-27 +date: 2026-09-01 maturity: draft editors: - name: Marcin Rataj @@ -134,23 +134,25 @@ order shown in the chosen variant below. :::warning The two orderings produce different bytes, and therefore different CIDs, for -the same logical node. Which ordering a writer emits is controlled by the -selected [profile](#profiles): +the same logical node: -- `Data`-first is written under the `unixfs-v1-2026` and later profiles. -- `Links`-first is the legacy ordering written under the `unixfs-v0-2015` and - `unixfs-v1-2025` profiles, and the canonical field order of the historical - [DAG-PB][ipld-dag-pb] codec specification. For UnixFS data, this document - takes precedence over the historical DAG-PB codec specification. +- `Links`-first is the canonical ordering. It is also the canonical field + order of the historical [DAG-PB][ipld-dag-pb] codec specification; for + UnixFS data, this document takes precedence over the historical DAG-PB + codec specification. +- `Data`-first is opt-in: no profile writes it. Implementations MAY expose an + explicit setting for writers that need it; enabling it changes the CID of + every written node. Readers SHOULD accept both orderings. Writers SHOULD support both orderings -and write the ordering mandated by the selected profile. Specialized -implementations MAY support a single ordering, for example a streaming-oriented -producer that only emits `Data`-first. Implementations that interoperate with -content on the public IPFS Mainnet MUST accept both orderings when reading. +and SHOULD write `Links`-first unless the user explicitly opted into +`Data`-first. Specialized implementations MAY support a single ordering, for +example a streaming-oriented producer that only emits `Data`-first. +Implementations that interoperate with content on the public IPFS Mainnet +MUST accept both orderings when reading. ::: -`Data`-first ordering, written under the `unixfs-v1-2026` and later profiles: +`Data`-first ordering (opt-in): ```protobuf message PBNode { @@ -162,8 +164,7 @@ message PBNode { } ``` -`Links`-first ordering (legacy), written under the `unixfs-v0-2015` and -`unixfs-v1-2025` profiles: +`Links`-first ordering (canonical): ```protobuf message PBNode { @@ -178,7 +179,7 @@ message PBNode { The `Data`-first ordering lets a streaming reader process the UnixFS metadata in `Data` (for example, HAMTShard `hashType` and `fanout`) before the links, and stop reading links early once it finds the entry it is looking for. The -legacy ordering keeps the CIDs of already-published content stable. +canonical ordering keeps the CIDs of already-published content stable. After decoding the node, we obtain a `PBNode`. This `PBNode` contains a field `Data` that contains the bytes that require the second decoding. This will also be @@ -777,8 +778,7 @@ The following profiles are defined: | Profile | Defined in | Description | | --- | --- | --- | | `unixfs-v0-2015` | [IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/) | Legacy CIDv0 parameters matching Kubo defaults through v0.39. For reproducing historical CIDv0 references. | -| `unixfs-v1-2025` | [IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/) | Deterministic CIDv1 parameters with modern settings. Writes the legacy `Links`-first `PBNode` ordering. | -| `unixfs-v1-2026` | [IPIP-0550](https://specs.ipfs.tech/ipips/ipip-0550/) | Same as `unixfs-v1-2025`, plus the `Data`-first `PBNode` ordering for more efficient streaming reads. | +| `unixfs-v1-2025` | [IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/) | Deterministic CIDv1 parameters with modern settings. | Implementations SHOULD use these exact profile names when exposing profile selection in configuration, command-line flags, documentation, and test @@ -1037,8 +1037,8 @@ Test vectors for UnixFS directory structures, progressing from simple flat direc [`dag-pb` HAMTDirectory](#dag-pb-hamtdirectory) in both `PBNode` field orderings (see [`dag-pb` Node](#dag-pb-node) and [Profiles](#profiles)) - CIDs: - - `bafybeigqvyloizmfcdy6scaxnyltftzptaruqa3hnnplfzsbf4sqteiwlm`: `Directory`, `Data`-first (`unixfs-v1-2026`) - - `bafybeigdcg7pksx2zk5336vrfsktjodlr4rbfz37qr3koc5xboxe5ekv24`: `Directory`, `Links`-first (`unixfs-v1-2025`) + - `bafybeigqvyloizmfcdy6scaxnyltftzptaruqa3hnnplfzsbf4sqteiwlm`: `Directory`, `Data`-first (opt-in) + - `bafybeigdcg7pksx2zk5336vrfsktjodlr4rbfz37qr3koc5xboxe5ekv24`: `Directory`, `Links`-first (canonical) - `bafybeicwgy2rlqmqqu3yy2tqvm2wbgdvy3snu4sbbv4wqpvpnoplpzxz74`: `HAMTShard`, `Data`-first (hand-crafted) - `bafybeicjwkfslu7gwyywffvqgse5kiibojtktxcdqhgv7ldj5fjdacuceq`: `HAMTShard`, `Links`-first (hand-crafted) - Contents: each root holds a single `hello.txt` ("hello\n") stored as a From 30f42c75b7cd6794f67fecdab33d4f93cdf04d20 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Wed, 2 Sep 2026 18:10:42 +0200 Subject: [PATCH 10/14] docs: read-side motivation and credits Motivation now spells out the primary goal: both orderings exist in the wild, reading either is the guarantee, and the opt-in writer keeps that read path exercised. Cites the robustness principle from the architecture specs. --- src/ipips/ipip-0550.md | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/src/ipips/ipip-0550.md b/src/ipips/ipip-0550.md index ca4bcb56..2776e55c 100644 --- a/src/ipips/ipip-0550.md +++ b/src/ipips/ipip-0550.md @@ -1,6 +1,6 @@ --- title: "IPIP-0550: PBNode field ordering" -date: 2026-09-01 +date: 2026-09-02 ipip: proposal editors: - name: Alex Potsides @@ -9,8 +9,18 @@ editors: affiliation: name: Shipyard url: https://ipshipyard.com + - name: Marcin Rataj + github: lidel + affiliation: + name: Shipyard + url: https://ipshipyard.com/ relatedIssues: - https://github.com/ipfs/specs/issues/533 +thanks: + - name: Rod Vagg + github: rvagg + - name: Volker Mische + github: vmx order: 550 tags: ['ipips'] --- @@ -46,6 +56,29 @@ When the reader is attempting to traverse to a single entry deep in the shard, they are forced to read all entries for the current sub-shard before they can move deeper within the shard, which leads to inefficient traversals. +Independent of any efficiency gains, both orderings already exist in the +wild: the DAG-PB codec specification requires decoders to accept either +order "as IPFS data exists in both forms". Implementations comply, but by +accident. Neither the UnixFS specification nor any test suite said so, and +tolerance that is unspecified and untested is how interoperability decays: +the code path works until the day it silently does not, and nothing catches +it. + +The primary goal of this IPIP is the read side: every implementation can +read UnixFS DAGs in either field order, and fixtures and conformance tests +prove it. Producing the non-canonical `Data`-first order is optional and +opt-in, but keeping a conforming writer around matters too: it is what keeps +the tolerant read path exercised, so reading and writing such DAGs both stay +interoperable instead of becoming an untested promise. + +This is the +[robustness principle](https://specs.ipfs.tech/architecture/principles/#robustness) +applied: "Be strict about the outcomes, be tolerant about the methods". The +outcome is strict: the same logical UnixFS DAG, pinned by byte-exact +fixtures and CIDs. The method, the field order on the wire, is tolerated in +both forms, and, per the same principle's warning about silent tolerance, +kept honest with tests rather than left to accident. + ## Detailed design If content authors are allowed to write the `Data` field first, readers can From ddc147cb99c46eb535a66bc70ec81ccd94ea87dd Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 7 Sep 2026 16:50:17 +0200 Subject: [PATCH 11/14] docs: dag-pb spec now permits both orders ipld/ipld#383 merged reworked: encoders may write either order, decoders accept both, default unchanged. Compatibility drops the stale precedence claim, Alternatives records the rework, working code notes merges and the @ipld/dag-pb v4.2.0 release. --- src/ipips/ipip-0550.md | 33 +++++++++++++++++---------------- src/unixfs.md | 8 ++++---- 2 files changed, 21 insertions(+), 20 deletions(-) diff --git a/src/ipips/ipip-0550.md b/src/ipips/ipip-0550.md index 2776e55c..b1fefff0 100644 --- a/src/ipips/ipip-0550.md +++ b/src/ipips/ipip-0550.md @@ -1,6 +1,6 @@ --- title: "IPIP-0550: PBNode field ordering" -date: 2026-09-02 +date: 2026-09-07 ipip: proposal editors: - name: Alex Potsides @@ -135,8 +135,11 @@ Traversing HAMT shards is more expensive than it needs to be, which disproportionately affects resource-constrained environments and inefficient runtimes. -Working code: opt-in `data-first` writing in boxo ([ipfs/boxo#1212](https://github.com/ipfs/boxo/pull/1212)) -and Kubo ([ipfs/kubo#11439](https://github.com/ipfs/kubo/pull/11439)); +Working code: opt-in `data-first` writing merged in boxo +([ipfs/boxo#1212](https://github.com/ipfs/boxo/pull/1212)) and Kubo +([ipfs/kubo#11439](https://github.com/ipfs/kubo/pull/11439)), and shipped in +[@ipld/dag-pb v4.2.0](https://github.com/ipld/js-dag-pb/releases/tag/v4.2.0) +([ipld/js-dag-pb#111](https://github.com/ipld/js-dag-pb/pull/111)); cross-ordering read tests and CAR fixtures shipped in [gateway-conformance v0.14.1](https://github.com/ipfs/gateway-conformance/releases/tag/v0.14.1). @@ -154,15 +157,12 @@ developments that want the more efficient streaming reads. `PBNode` wire ordering is governed by the [DAG-PB codec specification](https://ipld.io/specs/codecs/dag-pb/spec/#protobuf-strictness), which requires decoders to accept both field orders ("as IPFS data exists in -both forms") while mandating `Links`-first on encode. Deployed implementations +both forms") and, since [ipld/ipld#383](https://github.com/ipld/ipld/pull/383), +explicitly permits encoders to write either order. Deployed implementations (go-codec-dagpb, js-dag-pb) already read both orders, so `Data`-first blocks -are readable by existing software. - -On the write side, `Data`-first deviates from the canonical encode order of the -historical DAG-PB codec specification. For UnixFS data, the -[UnixFS specification](https://specs.ipfs.tech/unixfs/) takes precedence over -the historical DAG-PB codec specification: blocks produced with the opt-in -`data-first` setting are valid UnixFS. +are readable by existing software, and the two specifications agree. Should +they diverge again, the [UnixFS specification](https://specs.ipfs.tech/unixfs/) +governs UnixFS data. Backward compatibility is preserved by keeping the new ordering opt-in. Writers emit it only when a user explicitly enables the setting; every profile and @@ -190,11 +190,12 @@ Two valid encodings for the same data are possible. producing different CIDs for the same input, breaking backward compatibility and the CID determinism that `unixfs-v1-2025` ([IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/)) guarantees. -- **Change the canonical encode order in the DAG-PB codec specification** - ([ipld/ipld#383](https://github.com/ipld/ipld/pull/383)). This has the same - effect as changing the ordering unconditionally: writers following the - updated codec specification would silently produce new CIDs for existing - content. Ordering must remain an opt-in choice at the UnixFS layer. +- **Flip the canonical encode order in the DAG-PB codec specification**, as + first proposed in [ipld/ipld#383](https://github.com/ipld/ipld/pull/383). + This has the same effect as changing the ordering unconditionally: writers + following the updated codec specification would silently produce new CIDs + for existing content. That PR was reworked to permit both orders without + changing the default, and merged in that form. - **Reader-side optimization without a format change.** `Links` fields are length-delimited, so a reader holding a complete block can skip them cheaply and read `Data` at the tail. This helps whole-block parsing, but does not diff --git a/src/unixfs.md b/src/unixfs.md index c5353ef9..83487602 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -136,10 +136,10 @@ order shown in the chosen variant below. The two orderings produce different bytes, and therefore different CIDs, for the same logical node: -- `Links`-first is the canonical ordering. It is also the canonical field - order of the historical [DAG-PB][ipld-dag-pb] codec specification; for - UnixFS data, this document takes precedence over the historical DAG-PB - codec specification. +- `Links`-first is the canonical ordering for UnixFS data. The + [DAG-PB][ipld-dag-pb] codec specification permits either order on encode + and requires decoders to accept both; where the two documents diverge, + this one governs UnixFS data. - `Data`-first is opt-in: no profile writes it. Implementations MAY expose an explicit setting for writers that need it; enabling it changes the CID of every written node. From 37f2e9d0d5e8a6eaa3fe817e6dec74ba8051f1a2 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 7 Sep 2026 16:50:17 +0200 Subject: [PATCH 12/14] chore: update status --- src/ipips/ipip-0550.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ipips/ipip-0550.md b/src/ipips/ipip-0550.md index b1fefff0..6a542592 100644 --- a/src/ipips/ipip-0550.md +++ b/src/ipips/ipip-0550.md @@ -1,7 +1,7 @@ --- title: "IPIP-0550: PBNode field ordering" date: 2026-09-07 -ipip: proposal +ipip: ratified editors: - name: Alex Potsides github: achingbrain From 06346b979a97b8399c36681a2f4a323fd561ab5e Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 7 Sep 2026 17:03:43 +0200 Subject: [PATCH 13/14] chore: bump unixfs.md date --- src/unixfs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/unixfs.md b/src/unixfs.md index 83487602..0d5ab1b9 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -3,7 +3,7 @@ title: UnixFS description: > UnixFS is a Protocol Buffers-based format for describing files and directories as dag-pb DAGs and raw blocks in IPFS. -date: 2026-09-01 +date: 2026-09-07 maturity: draft editors: - name: Marcin Rataj From 4f5d2d4e90e7ab59452cd76bc3255e000a0911a7 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 7 Sep 2026 17:29:05 +0200 Subject: [PATCH 14/14] docs: list PBNode field order in IPIP-0499 parameters --- src/ipips/ipip-0499.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/ipips/ipip-0499.md b/src/ipips/ipip-0499.md index 31540246..4e11ff2d 100644 --- a/src/ipips/ipip-0499.md +++ b/src/ipips/ipip-0499.md @@ -91,6 +91,7 @@ The following [UnixFS](https://specs.ipfs.tech/unixfs/) parameters were identifi 1. [Symlink](https://specs.ipfs.tech/unixfs/#dag-pb-symlink) handling: preserved as UnixFS Type=4 nodes, or followed (dereferenced to target). 1. [Mode](https://specs.ipfs.tech/unixfs/#mode-field): optional POSIX file permissions. 1. [Mtime](https://specs.ipfs.tech/unixfs/#mtime-field): optional modification timestamp. +1. [PBNode field order](https://specs.ipfs.tech/unixfs/#dag-pb-node): `links-first` (canonical) or `data-first` (opt-in, see [IPIP-0550](https://specs.ipfs.tech/ipips/ipip-0550/)). ### Balanced DAG layout variants