diff --git a/src/ipips/ipip-0499.md b/src/ipips/ipip-0499.md index eb94da2d..4e11ff2d 100644 --- a/src/ipips/ipip-0499.md +++ b/src/ipips/ipip-0499.md @@ -91,6 +91,7 @@ The following [UnixFS](https://specs.ipfs.tech/unixfs/) parameters were identifi 1. [Symlink](https://specs.ipfs.tech/unixfs/#dag-pb-symlink) handling: preserved as UnixFS Type=4 nodes, or followed (dereferenced to target). 1. [Mode](https://specs.ipfs.tech/unixfs/#mode-field): optional POSIX file permissions. 1. [Mtime](https://specs.ipfs.tech/unixfs/#mtime-field): optional modification timestamp. +1. [PBNode field order](https://specs.ipfs.tech/unixfs/#dag-pb-node): `links-first` (canonical) or `data-first` (opt-in, see [IPIP-0550](https://specs.ipfs.tech/ipips/ipip-0550/)). ### Balanced DAG layout variants @@ -214,6 +215,7 @@ Based on the research above, we define **`unixfs-v1-2025`** as an opinionated pr | Symlinks | preserved | | Mode (permissions) | excluded (opt-in) | | Mtime (modification time) | excluded (opt-in) | +| PBNode field order | links-first | ### The `unixfs-v0-2015` legacy profile @@ -238,6 +240,7 @@ Note: this profile is a best-effort approximation of historical behavior. It pro | Symlinks | preserved | | Mode (permissions) | excluded (opt-in) | | Mtime (modification time) | excluded (opt-in) | +| PBNode field order | links-first | ## Design rationale diff --git a/src/ipips/ipip-0550.md b/src/ipips/ipip-0550.md new file mode 100644 index 00000000..6a542592 --- /dev/null +++ b/src/ipips/ipip-0550.md @@ -0,0 +1,242 @@ +--- +title: "IPIP-0550: PBNode field ordering" +date: 2026-09-07 +ipip: ratified +editors: + - name: Alex Potsides + github: achingbrain + url: https://achingbrain.net/ + affiliation: + name: Shipyard + url: https://ipshipyard.com + - name: Marcin Rataj + github: lidel + affiliation: + name: Shipyard + url: https://ipshipyard.com/ +relatedIssues: + - https://github.com/ipfs/specs/issues/533 +thanks: + - name: Rod Vagg + github: rvagg + - name: Volker Mische + github: vmx +order: 550 +tags: ['ipips'] +--- + +## Summary + +Add an opt-in `Data`-first field ordering for `PBNode` protobuf messages, so +that streaming readers can process UnixFS metadata before links, and +formalize that readers accept both orderings. No profile enables it: all +profiles keep the canonical `Links`-first ordering and their CIDs, and +writing `Data`-first requires an explicit opt-in setting. + +## Motivation + +Regular UnixFS and HAMT-sharded directories are encoded as `PBNode` protobuf +messages. + +HAMT-sharded directory entries have the characteristic of prefixing the name of +each entry with a number of characters drawn from the hash of the directory +entry name. + +When multiple entries land in the same bucket, that bucket is replaced by a +sub-shard with its own CID/block that contains those entries. + +The settings used to derive the prefix characters are stored in the `Data` +field of the `PBNode` protobuf message. + +This means that all `Link` messages must be read from the `PBNode` message +before we can read the `hashType` (a multihash code) and `fanout` values that +let us calculate the prefix length for a given directory entry. + +When the reader is attempting to traverse to a single entry deep in the shard, +they are forced to read all entries for the current sub-shard before they can +move deeper within the shard, which leads to inefficient traversals. + +Independent of any efficiency gains, both orderings already exist in the +wild: the DAG-PB codec specification requires decoders to accept either +order "as IPFS data exists in both forms". Implementations comply, but by +accident. Neither the UnixFS specification nor any test suite said so, and +tolerance that is unspecified and untested is how interoperability decays: +the code path works until the day it silently does not, and nothing catches +it. + +The primary goal of this IPIP is the read side: every implementation can +read UnixFS DAGs in either field order, and fixtures and conformance tests +prove it. Producing the non-canonical `Data`-first order is optional and +opt-in, but keeping a conforming writer around matters too: it is what keeps +the tolerant read path exercised, so reading and writing such DAGs both stay +interoperable instead of becoming an untested promise. + +This is the +[robustness principle](https://specs.ipfs.tech/architecture/principles/#robustness) +applied: "Be strict about the outcomes, be tolerant about the methods". The +outcome is strict: the same logical UnixFS DAG, pinned by byte-exact +fixtures and CIDs. The method, the field order on the wire, is tolerated in +both forms, and, per the same principle's warning about silent tolerance, +kept honest with tests rather than left to accident. + +## Detailed design + +If content authors are allowed to write the `Data` field first, readers can +process `PBNode` messages with a streaming parser: the UnixFS metadata (for +HAMT shards, the `hashType` and `fanout` parameters) arrives before the links, +so a reader looking for a specific entry can stop reading links as soon as it +finds the one it needs. + +### The `PBNode field order` parameter + +Writing the `Data` field first changes the CID generated for a piece of +content, so this is an opt-in change. + +We add a new parameter to the set defined in +[IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/): + +| Parameter | Values | +| ------------------ | --------------------------------------- | +| PBNode field order | `links-first` (default) or `data-first` | + +When a profile does not define `PBNode field order`, it is `links-first`. +This IPIP amends the IPIP-0499 profile tables so `unixfs-v0-2015` and +`unixfs-v1-2025` state `links-first` explicitly; they continue to produce the +same CIDs as today. + +No profile sets `data-first` (see [Alternatives](#alternatives) for why). +Implementations MAY expose an explicit low-level opt-in setting for writers +that need it; enabling it changes the CID of every written node. + +### Changes to existing specifications + +This IPIP amends the [UnixFS specification](https://specs.ipfs.tech/unixfs/): + +- documents both `PBNode` field orderings: `Links`-first as the canonical + form, `Data`-first as the explicit opt-in +- adds ordering requirements: readers SHOULD accept both orderings, and + implementations that interoperate with content on the public IPFS Mainnet + MUST accept both when reading; writers SHOULD support both orderings and + SHOULD write `Links`-first unless the user explicitly opted into + `Data`-first; specialized implementations MAY support a single ordering +- adds a [Profiles](https://specs.ipfs.tech/unixfs/#profiles) section: a + registry of profile names that implementations SHOULD use in configuration, + flags, and test suites +- adds test vectors covering both orderings + +It also amends the profile tables in +[IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/) with an explicit +`PBNode field order: links-first` row, so the preexisting profiles are +protected from unintentional change. + +## Design rationale + +Traversing HAMT shards is more expensive than it needs to be, which +disproportionately affects resource-constrained environments and inefficient +runtimes. + +Working code: opt-in `data-first` writing merged in boxo +([ipfs/boxo#1212](https://github.com/ipfs/boxo/pull/1212)) and Kubo +([ipfs/kubo#11439](https://github.com/ipfs/kubo/pull/11439)), and shipped in +[@ipld/dag-pb v4.2.0](https://github.com/ipld/js-dag-pb/releases/tag/v4.2.0) +([ipld/js-dag-pb#111](https://github.com/ipld/js-dag-pb/pull/111)); +cross-ordering read tests and CAR fixtures shipped in +[gateway-conformance v0.14.1](https://github.com/ipfs/gateway-conformance/releases/tag/v0.14.1). + +### User benefit + +Traversing HAMT shards will become faster in resource-constrained environments +and inefficient runtimes. + +Existing content and workflows are unaffected: the canonical ordering remains +the default everywhere, and an explicit opt-in is available for new +developments that want the more efficient streaming reads. + +### Compatibility + +`PBNode` wire ordering is governed by the +[DAG-PB codec specification](https://ipld.io/specs/codecs/dag-pb/spec/#protobuf-strictness), +which requires decoders to accept both field orders ("as IPFS data exists in +both forms") and, since [ipld/ipld#383](https://github.com/ipld/ipld/pull/383), +explicitly permits encoders to write either order. Deployed implementations +(go-codec-dagpb, js-dag-pb) already read both orders, so `Data`-first blocks +are readable by existing software, and the two specifications agree. Should +they diverge again, the [UnixFS specification](https://specs.ipfs.tech/unixfs/) +governs UnixFS data. + +Backward compatibility is preserved by keeping the new ordering opt-in. Writers +emit it only when a user explicitly enables the setting; every profile and +default keeps the canonical ordering, so already-published CIDs and the CID +determinism guarantees of `unixfs-v1-2025` are unchanged. + +### Security + +Two valid encodings for the same data are possible. + +- Same content, two CIDs, which harms deduplication, and CID-based denylists + need to include both CIDs. +- Round trips from blocks to data and back change CIDs whenever the writer's + ordering differs from the ordering of the original block, in either + direction: a canonical writer re-encoding a `data-first` block, or a + `data-first` writer re-encoding a canonical block. + - This can reveal history of writes in systems like MFS. +- Streaming readers MUST NOT act on links, for example by fetching child + blocks, before the enclosing block has been fully received and its multihash + verified. + +### Alternatives + +- **Change the ordering unconditionally.** Every implementation would start + producing different CIDs for the same input, breaking backward compatibility + and the CID determinism that `unixfs-v1-2025` + ([IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/)) guarantees. +- **Flip the canonical encode order in the DAG-PB codec specification**, as + first proposed in [ipld/ipld#383](https://github.com/ipld/ipld/pull/383). + This has the same effect as changing the ordering unconditionally: writers + following the updated codec specification would silently produce new CIDs + for existing content. That PR was reworked to permit both orders without + changing the default, and merged in that form. +- **Reader-side optimization without a format change.** `Links` fields are + length-delimited, so a reader holding a complete block can skip them cheaply + and read `Data` at the tail. This helps whole-block parsing, but does not + help streaming parsers, which cannot skip ahead in data that has not arrived + yet. +- **Ship a `unixfs-v1-YYYY` profile that enables `data-first`.** Considered + and dropped: a dated successor to `unixfs-v1-2025` reads as "the + recommended latest", inviting unintentional adoption and a de facto new + CIDv1 default, exactly the CID churn this IPIP avoids. Creating a new CID + profile should be a separate IPIP, and its review should take a long time: + a new dated profile effectively informs the new suggested default for + CIDv1, and this IPIP deliberately changes no defaults. The parameter + machinery stays, so such a future IPIP can define the profile if + multi-implementation demand for deterministic `data-first` CIDs appears. + +## Test fixtures + +The table below lists minimal single-block vectors for both orderings. To +verify compliance, decode each block, confirm the CID, and confirm that +re-encoding the decoded node with the stated ordering reproduces the block +bytes. Readers MUST resolve `hello.txt` through all four directory roots. + +| Description | Ordering | CID | Block (base16 encoded) | +| --- | --- | --- | --- | +| `hello.txt`, file content "hello\n" | raw leaf | `bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am` | `68656c6c6f0a` | +| `Directory` containing `hello.txt` | `Data`-first (opt-in) | `bafybeigqvyloizmfcdy6scaxnyltftzptaruqa3hnnplfzsbf4sqteiwlm` | `0a02080112330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e7478741806` | +| `Directory` containing `hello.txt` | `Links`-first (canonical) | `bafybeigdcg7pksx2zk5336vrfsktjodlr4rbfz37qr3koc5xboxe5ekv24` | `12330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e74787418060a020801` | +| `HAMTShard` containing `hello.txt` | `Data`-first (hand-crafted) | `bafybeicwgy2rlqmqqu3yy2tqvm2wbgdvy3snu4sbbv4wqpvpnoplpzxz74` | `0a250805121c80000000000000000000000000000000000000000000000000000000282230800212350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e7478741806` | +| `HAMTShard` containing `hello.txt` | `Links`-first (hand-crafted) | `bafybeicjwkfslu7gwyywffvqgse5kiibojtktxcdqhgv7ldj5fjdacuceq` | `12350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e74787418060a250805121c800000000000000000000000000000000000000000000000000000002822308002` | + +The `HAMTShard` blocks are hand-crafted for parser testing; no profile shards a +single-entry directory via import (sharding starts above the 256 KiB +threshold). The entry name `hello.txt` hashed with murmur3-x64-64 (`hashType` +`0x22`) yields `0xDF` as the first byte, selecting bucket 223 at `fanout` 256: +the link name is `DF` + `hello.txt`, and bit 223 is set in the +`decode(PBNode.Data).Data` bitfield. + +All five blocks ship as +[`pbnode-field-orders.car` in gateway-conformance v0.14.1](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.14.1/fixtures/path_gateway_unixfs/pbnode-field-orders.car), +whose conformance tests exercise both orderings. + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). diff --git a/src/unixfs.md b/src/unixfs.md index 2598700c..0d5ab1b9 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -3,7 +3,7 @@ title: UnixFS description: > UnixFS is a Protocol Buffers-based format for describing files and directories as dag-pb DAGs and raw blocks in IPFS. -date: 2026-08-22 +date: 2026-09-07 maturity: draft editors: - name: Marcin Rataj @@ -125,7 +125,48 @@ message PBLink { // cumulative size of target object uint64 Tsize = 3; } +``` + +The `PBNode` message holds two fields, `Data` (field number 1) and `Links` +(field number 2). Two wire orderings of these fields exist, and both decode to +the same logical node. A conforming writer emits the fields in the declaration +order shown in the chosen variant below. + +:::warning +The two orderings produce different bytes, and therefore different CIDs, for +the same logical node: + +- `Links`-first is the canonical ordering for UnixFS data. The + [DAG-PB][ipld-dag-pb] codec specification permits either order on encode + and requires decoders to accept both; where the two documents diverge, + this one governs UnixFS data. +- `Data`-first is opt-in: no profile writes it. Implementations MAY expose an + explicit setting for writers that need it; enabling it changes the CID of + every written node. + +Readers SHOULD accept both orderings. Writers SHOULD support both orderings +and SHOULD write `Links`-first unless the user explicitly opted into +`Data`-first. Specialized implementations MAY support a single ordering, for +example a streaming-oriented producer that only emits `Data`-first. +Implementations that interoperate with content on the public IPFS Mainnet +MUST accept both orderings when reading. +::: + +`Data`-first ordering (opt-in): +```protobuf +message PBNode { + // opaque user data + bytes Data = 1; + + // refs to other objects + repeated PBLink Links = 2; +} +``` + +`Links`-first ordering (canonical): + +```protobuf message PBNode { // refs to other objects repeated PBLink Links = 2; @@ -135,6 +176,11 @@ message PBNode { } ``` +The `Data`-first ordering lets a streaming reader process the UnixFS metadata +in `Data` (for example, HAMTShard `hashType` and `fanout`) before the links, +and stop reading links early once it finds the entry it is looking for. The +canonical ordering keeps the CIDs of already-published content stable. + After decoding the node, we obtain a `PBNode`. This `PBNode` contains a field `Data` that contains the bytes that require the second decoding. This will also be a protobuf message specified in the UnixFSV1 format: @@ -368,12 +414,12 @@ The HAMT directory is configured through the UnixFS metadata in `PBNode.Data`: - `decode(PBNode.Data).fanout` is REQUIRED for HAMTShard nodes (though marked optional in the protobuf schema). The value MUST be a power of two, a multiple of 8 (for byte-aligned bitfields), and at most 1024. - + This determines the number of possible bucket indices (permutations) at each level of the trie. For example, fanout=256 provides 256 possible buckets (0x00 to 0xFF), requiring 8 bits from the hash. The hex prefix length is `log2(fanout)/4` characters (since each hex character represents 4 bits). The same fanout value is used throughout all levels of a single HAMT structure - + :::note Implementations that onboard user data to create new HAMTDirectory structures are free to choose a `fanout` value or allow users to configure it based on their use case: - **256**: Balanced tree depth and node size, suitable for most use cases @@ -382,7 +428,7 @@ The HAMT directory is configured through the UnixFS metadata in `PBNode.Data`: - Trade-offs: Larger blocks mean higher latency on cold cache reads and more data rewritten when modifying directories (each change affects a larger block) ::: - + :::warning Implementations MUST limit the `fanout` parameter to a maximum of 1024 to prevent denial-of-service attacks. Excessively large fanout values can cause memory exhaustion @@ -719,6 +765,25 @@ The following names SHOULD NOT be used in UnixFS directories: contain `/` (see [Paths](#paths)): a directory entry with such a name cannot be addressed by any UnixFS path and is reachable only by its own CID. +# Profiles + +DAG construction parameters such as chunk size, DAG width, HAMT fanout and +threshold, and `PBNode` field ordering all affect the resulting CID, so the +same input can produce different CIDs across implementations. A :dfn[Profile] +is a named, complete set of these parameters: two implementations importing +the same input under the same profile produce the same CID. + +The following profiles are defined: + +| Profile | Defined in | Description | +| --- | --- | --- | +| `unixfs-v0-2015` | [IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/) | Legacy CIDv0 parameters matching Kubo defaults through v0.39. For reproducing historical CIDv0 references. | +| `unixfs-v1-2025` | [IPIP-0499](https://specs.ipfs.tech/ipips/ipip-0499/) | Deterministic CIDv1 parameters with modern settings. | + +Implementations SHOULD use these exact profile names when exposing profile +selection in configuration, command-line flags, documentation, and test +suites, so that a profile name means the same thing in every tool. + # Appendix: Test Vectors :::warning @@ -965,6 +1030,25 @@ Test vectors for UnixFS directory structures, progressing from simple flat direc - Link Names in HAMT have 2-character hex prefix (hash buckets) - Can retrieve any file by name through hash bucket calculation +### PBNode Field Ordering + +- Fixture: [`pbnode-field-orders.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.14.1/fixtures/path_gateway_unixfs/pbnode-field-orders.car) + - Type: [`dag-pb` Directory](#dag-pb-directory) and + [`dag-pb` HAMTDirectory](#dag-pb-hamtdirectory) in both `PBNode` field + orderings (see [`dag-pb` Node](#dag-pb-node) and [Profiles](#profiles)) + - CIDs: + - `bafybeigqvyloizmfcdy6scaxnyltftzptaruqa3hnnplfzsbf4sqteiwlm`: `Directory`, `Data`-first (opt-in) + - `bafybeigdcg7pksx2zk5336vrfsktjodlr4rbfz37qr3koc5xboxe5ekv24`: `Directory`, `Links`-first (canonical) + - `bafybeicwgy2rlqmqqu3yy2tqvm2wbgdvy3snu4sbbv4wqpvpnoplpzxz74`: `HAMTShard`, `Data`-first (hand-crafted) + - `bafybeicjwkfslu7gwyywffvqgse5kiibojtktxcdqhgv7ldj5fjdacuceq`: `HAMTShard`, `Links`-first (hand-crafted) + - Contents: each root holds a single `hello.txt` ("hello\n") stored as a + `raw` leaf (`bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am`) + - Purpose: readers accept both `PBNode` field orderings + - Validation: + - `hello.txt` resolves through all four roots + - Byte-level vectors: fixtures table in + [IPIP-0550](https://specs.ipfs.tech/ipips/ipip-0550/) + ## Special Cases and Advanced Features Test vectors for special UnixFS features and edge cases.