feat: tests for IPIP-550 PBNode field ordering - #304
Merged
Conversation
Gateways must resolve UnixFS content regardless of PBNode field order: the legacy Links-first encoding and the Data-first order proposed for the unixfs-v1-2026 profile. - fixtures: pbnode-field-orders.car, single-entry Directory and HAMTShard in both orders, byte-identical to the IPIP table - tests: path resolution through all four roots, raw block responses byte-exact for all five blocks Refs ipfs/specs#550
Contributor
|
Results against Kubo latest: Summary
|
Contributor
|
Results against Kubo master: Summary
|
Collaborator
Author
|
All green, @achingbrain as expected reading dags in either order works with latext I'll ship this as a patch release as soon Kubo PR with CI that uses this branch passes green. |
lidel
added a commit
to ipfs/boxo
that referenced
this pull request
Aug 27, 2026
Temporary pin so the PBNode field ordering tests from ipfs/gateway-conformance#304 run against boxo gateway backends. Switch back to a tagged release once one ships.
This was referenced Aug 27, 2026
lidel
marked this pull request as ready for review
August 27, 2026 19:38
Contributor
v0.14.1Added
|
lidel
added a commit
to ipfs/boxo
that referenced
this pull request
Aug 27, 2026
v0.14.1 shipped the ipfs/gateway-conformance#304 tests, so the moving v0.14 tag covers them again.
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Aug 27, 2026
v0.14.1 shipped the ipfs/gateway-conformance#304 tests, so the moving v0.14 tag covers them again.
lidel
added a commit
to ipfs/boxo
that referenced
this pull request
Sep 4, 2026
* feat(unixfs): unixfs-v1-2026 profile (IPIP-550) Opt-in Data-first PBNode field ordering behind the new io.UnixFS_v1_2026 profile, per IPIP-550. Default output is unchanged: all existing profiles keep the legacy Links-first order and their CIDs. - ipld/merkledag: PBNodeFieldOrder global and a Data-first encoder used when a profile opts in (candidate for upstreaming to go-codec-dagpb) - ipld/unixfs/io: PBNodeFieldOrder profile parameter and UnixFS_v1_2026, wired through ApplyGlobals - tests assert byte-exact fixtures from the IPIP table Refs ipfs/specs#550 * ci: run gateway-conformance from ipip-550 branch Temporary pin so the PBNode field ordering tests from ipfs/gateway-conformance#304 run against boxo gateway backends. Switch back to a tagged release once one ships. * ci: gateway-conformance back to v0.14 v0.14.1 shipped the ipfs/gateway-conformance#304 tests, so the moving v0.14 tag covers them again. * fix(merkledag): rotate dagpb output for data-first Derive PBNodeDataFirst bytes from dagpb.AppendEncode by moving the trailing Data field to the front, instead of a second hand-written encoder. One encoder owns link sorting and field presence, so a decoded block with unsorted links now re-encodes sorted under both orders, and byte parity with links-first holds by construction. Unknown PBNodeFieldOrder values return an error instead of silently encoding links-first. - fieldorder.go: moveDataFirst; godoc spells out the process-wide nature of the setting, the set-once-at-startup constraint, and cites the DAG-PB strictness section - coding.go: switch on the order after AppendEncode, error on unknown - fieldorder_test.go: unknown-order test; property test over random nodes against a protowire-based links-first oracle, covering unsorted links, nil and empty Data, multi-byte length prefixes, and CIDv0, identity, and CIDv1 link hashes * test(unixfs): pin unixfs-v1-2026 CIDs end to end Build a three-chunk file and a sharded directory from fixed inputs under UnixFS_v1_2025 and UnixFS_v1_2026 and pin the root CIDs; the file CIDs match `ipfs add --chunker=size-1000` output under each profile. Every dag-pb block in both DAGs must lead with the profile's first field, and the two DAGs must decode to the same nodes. - profile.go: UnixFS_v1_2026 written as a full literal so the test asserts each parameter instead of reconstructing the copy - profile_test.go: per-field asserts for all three profiles; ApplyGlobals subtest checks all six globals; saveAndRestoreGlobals also restores chunk.DefaultBlockSize and helpers.DefaultLinksPerBlock so applied profiles no longer leak into later tests * docs(unixfs): which CIDs unixfs-v1-2026 changes Every dag-pb node with both Data and Links gets a new CID, files larger than one chunk included; single-chunk raw-leaf files keep theirs. Existing links-first directories are re-encoded when reopened through the directory API (MFS directories on their next access), a sharded root first and each child shard as it is loaded. The godocs also state why the setting is a process-wide global and that it must be applied once at startup. - profile.go: field, profile and ApplyGlobals godoc - doc.go: UnixFS_v1_2026 in the profile list, Global Settings section - CHANGELOG.md: scope, MFS re-encode, ✨ marker, PR link * style(unixfs): gofumpt profile_test.go * refactor: drop UnixFS_v1_2026, keep opt-in knob A dated successor profile invites unintentional adoption and a de facto new CIDv1 default. PBNodeFieldOrder stays as a documented low-level opt-in; UnixFS_v0_2015 and UnixFS_v1_2025 now pin PBNodeLinksFirst explicitly. Tests derive data-first from UnixFS_v1_2025 plus the knob and keep asserting the same IPIP-550 fixture bytes and CIDs. Refs ipfs/specs#550
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Sep 4, 2026
* feat: opt-in unixfs-v1-2026 profile (IPIP-550) Opt-in Data-first PBNode field ordering via the new unixfs-v1-2026 config profile, per IPIP-550. Defaults and the preexisting unixfs-v0-2015 and unixfs-v1-2025 profiles are unchanged and keep their CIDs. - config: Import.UnixFSPBNodeFieldOrder (links-first default, data-first) and the unixfs-v1-2026 profile applying unixfs-v1-2025 settings plus data-first - core/node: wires merkledag.DefaultPBNodeFieldOrder from config - deps: boxo bump to the ipfs/boxo#1212 encoder commit - test/cli: byte-exact fixtures from the IPIP-550 table; pinned CIDs for existing profiles unchanged Refs ipfs/specs#550 * ci: run gateway-conformance from ipip-550 commit Temporary pin to the ipfs/gateway-conformance#304 head so the PBNode field ordering tests run against kubo. Switch back to a tagged release once one ships. * ci: gateway-conformance back to v0.14 v0.14.1 shipped the ipfs/gateway-conformance#304 tests, so the moving v0.14 tag covers them again. * docs: unixfs-v1-2026 scope and MFS re-encode State what the profile actually changes: every dag-pb node with both Data and Links gets a new CID, files larger than one chunk included, and data already in MFS is upgraded to the new order on first read (plain `ipfs files ls` or `stat` included), a sharded root before its child shards; the MFS root is re-encoded by any command that starts a node. - config/profile.go, docs/config.md: profile description and the unixfs-v1-2026 section - docs/changelogs/v0.43.md: highlight leads with the need (readers get the HAMT layout before links) and the upgrade-on-first-use behavior * chore: update boxo to 04a079ec27b1 Pins the ipfs/boxo#1212 branch tip: data-first bytes derive from dagpb.AppendEncode so link sorting is inherited, unknown field order values return an error, and pinned end-to-end CIDs cover the profile. * refactor: drop unixfs-v1-2026, keep opt-in knob A dated successor profile invites unintentional adoption and a de facto new CIDv1 default. Import.UnixFSPBNodeFieldOrder stays as the documented low-level opt-in; unixfs-v0-2015 and unixfs-v1-2025 now pin links-first explicitly. - deps: boxo bump to the ipfs/boxo#1212 commit that drops UnixFS_v1_2026 - test/cli: field order exercised via the config knob, same IPIP-550 fixture bytes and CIDs Refs ipfs/specs#550 * chore: update boxo to 02026ddcf262 Squash-merge of ipfs/boxo#1212 on main. * docs: tighten v0.43.1 changelog - tighter CID profile entry; MFS caveat now says re-encode happens on rewrite, not read - deps: boxo pseudo-version with ipfs/boxo#1212, gateway-conformance v0.14.1 * docs: tighten Import.UnixFSPBNodeFieldOrder docs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Need
ipfs/specs#550 (IPIP-550) proposes an opt-in
Data-before-LinksPBNode field order (unixfs-v1-2026profile) so streaming readers get HAMT parameters before links. DAG-PB decoders already accept both orders1, but nothing in the suite checks that gateways actually serve content storedData-first, so a gateway failing on such blocks would go unnoticed.This PR
TestUnixFSPBNodeFieldOrder: resolveshello.txtthrough a UnixFSDirectoryand aHAMTShardin both field ordersTestTrustlessRawPBNodeFieldOrder: raw block responses return the stored bytes verbatim in both orders, no re-encoding to a preferred orderpbnode-field-orders.carfixture, byte-identical to the IPIP-550 fixtures tableiiuc this is READ-ONLY check, and it passes against current boxo/gateway, so is just a formality / future-proofing against regressions.
Write-side behavior (producing
Data-first blocks underunixfs-v1-2026) is out of scope here.Footnotes
DAG-PB spec, Protobuf Strictness, rule 2: decoders should accept either
PBNodefield order, "as IPFS data exists in both forms". ↩