Skip to content

Merge nested-object-end-marker - #48

Merged
AlexanderBuzz merged 3 commits into
masterfrom
nested-object-end-marker
Sep 2, 2026
Merged

Merge nested-object-end-marker#48
AlexanderBuzz merged 3 commits into
masterfrom
nested-object-end-marker

Conversation

@AlexanderBuzz

Copy link
Copy Markdown
Owner

Two pieces of work, both of them in the binary codec, released together as 2.4.0.

The nested ObjectEndMarker

StObject::fromJson() wrote the ObjectEndMarker once, after its loop, guarded
by whichever field instance the loop happened to leave behind. Fields are
serialized in canonical order, so a nested object is the last field of its parent
only as long as no field of a higher numbered type follows it — and where one
does, the marker was never written and the parser reads that following field back
as part of the nested object.

Transaction metadata is exactly that case on the XRP Ledger: every ModifiedNode
carrying both PreviousFields and FinalFields encoded the second inside the
first. fromParser() already wrote the marker per field; fromJson() now does
the same. Encodings that were correct before are unchanged, byte for byte.

Covered by testEncodeDecodeObjectFollowedByAnotherField and
testEncodeDecodeAdjacentObjects.

hardcastle/buffer 2.0

The constraint moves to ^2.0. 2.0 makes Buffer::$length private and ships six
behaviour fixes; each was checked against this codebase and none is reachable:

Change in Buffer 2.0 Here
$length no longer public not read anywhere
slice() past the end returns empty instead of throwing 22 call sites, all guarded by a length check or already ending in an exception
$buf[$i] = $x past the end is a no-op 11 indexed writes, 8 on plain arrays, 3 in bounds
unset($buf[$i]) is a no-op never used
toInt() throws beyond 8 bytes widest caller reads 4
concat($list, $totalLength) zero-fills no call passes a total length
out-of-range read*()/write*() throw Buffer's own accessors are never called; the readUInt8() here is BinaryParser's

One case did change behaviour, and it turned out to be a hole that predates the
upgrade: Ctid never validated the length of its input. Ctid::fromCtid('C002')
threw a ValueError out of slice() under 1.x and would now answer
getTransactionIndex() with 0. The constructor requires the sixteen
hexadecimal characters a CTID has, with a regression test.

Consumers reading ->length on a Buffer this library handed them need
getLength(), which has existed since Buffer 1.0 and works under both majors.

While in there

Several places concatenated buffers by unpacking them into int arrays,
array_merge()-ing those and rebuilding a buffer — work Buffer::concat() does
directly. BytesList::toBytes() is the one that matters, since every serialized
field passes through it and it merged in a loop. Amount::fromJson(),
AddressCodec::encodeXAddress() (which also went out through sprintf('%02X'),
join() and a hex decode) and BinaryParser::readUIntN() (which unpacked and
rebuilt the buffer it had just read) do the same thing on a smaller scale.
Encodings unchanged.

Buffer 2.0 types toArray() as list<int>, which made a cast in
BaseX::encode() redundant and Psalm fail the run; the cast is gone.

Verification

610 unit tests green, composer psalm exits 0.

3000 iterations over an IOU Payment, median of three runs:

buffer 1.0.1 buffer 2.0 2.0 + concat
encode 1005 ms 797 ms 780 ms
decode 1364 ms 908 ms 818 ms
encodeXAddress 305 ms 304 ms 280 ms

The string backing of Buffer 2.0 accounts for most of that; the concat changes
add roughly 10% on top for decoding.

StObject::fromJson() wrote the ObjectEndMarker once, after its loop, guarded by whichever field instance the loop had left behind. Fields are serialized in canonical order, so a nested object is the last field of its parent only as long as no field of a higher numbered type follows it - and where one does, the marker was never written and the parser reads that following field back as part of the nested object.

Transaction metadata is that case on the XRP Ledger: every ModifiedNode carrying both PreviousFields and FinalFields encoded the second inside the first. fromParser() already wrote the marker per field, and fromJson() now does the same.

Encodings that were correct before are unchanged, byte for byte. Covered by testEncodeDecodeObjectFollowedByAnotherField and testEncodeDecodeAdjacentObjects.
The constraint becomes ^2.0. That release makes Buffer::$length private and ships six behaviour fixes; each was checked against this codebase and none is reachable: $length is read nowhere, every slice() past the end of a buffer is guarded by a length check or already ends in an exception, the eleven indexed writes are either on plain arrays or in bounds, unset() on a buffer never happens, no toInt() sees more than four bytes, no concat() passes a total length, and Buffer's own read*()/write*() accessors are never called - the readUInt8() here is BinaryParser's.

One case did change behaviour, and the hole predates the upgrade: Ctid never validated the length of its input. Ctid::fromCtid('C002') threw a ValueError out of slice() under 1.x and would now answer getTransactionIndex() with 0. The constructor requires the sixteen hexadecimal characters a CTID has.

Several places concatenated buffers by unpacking them into int arrays, array_merge()-ing those and rebuilding a buffer - work Buffer::concat() does directly. BytesList::toBytes() is the one that matters, since every serialized field passes through it and it merged in a loop. Amount::fromJson(), AddressCodec::encodeXAddress(), which also went out through sprintf('%02X'), join() and a hex decode, and BinaryParser::readUIntN(), which unpacked and rebuilt the buffer it had just read, do the same on a smaller scale. Encodings are unchanged, byte for byte.

Buffer 2.0 types toArray() as list<int>, which made a cast in BaseX::encode() redundant and Psalm fail the run; the cast is gone.

3000 iterations over an IOU Payment, median of three runs, for buffer 1.0.1, buffer 2.0 and 2.0 with the concat changes: encode 1005 -> 797 -> 780 ms, decode 1364 -> 908 -> 818 ms, encodeXAddress 305 -> 304 -> 280 ms. The string backing of 2.0 accounts for most of it.
3.0.0 stays reserved for removing the Xahau types once hardcastle/xahau_php exists, so this goes out as a minor.

The one thing consumers have to know is in the entry: code of theirs that reads ->length on a Buffer this library handed it needs getLength(). That method has existed since Buffer 1.0, so the change is valid under both majors.
@AlexanderBuzz
AlexanderBuzz merged commit ff2947d into master Sep 2, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant