Merge nested-object-end-marker - #48
Merged
Merged
Conversation
StObject::fromJson() wrote the ObjectEndMarker once, after its loop, guarded by whichever field instance the loop had left behind. Fields are serialized in canonical order, so a nested object is the last field of its parent only as long as no field of a higher numbered type follows it - and where one does, the marker was never written and the parser reads that following field back as part of the nested object. Transaction metadata is that case on the XRP Ledger: every ModifiedNode carrying both PreviousFields and FinalFields encoded the second inside the first. fromParser() already wrote the marker per field, and fromJson() now does the same. Encodings that were correct before are unchanged, byte for byte. Covered by testEncodeDecodeObjectFollowedByAnotherField and testEncodeDecodeAdjacentObjects.
The constraint becomes ^2.0. That release makes Buffer::$length private and ships six behaviour fixes; each was checked against this codebase and none is reachable: $length is read nowhere, every slice() past the end of a buffer is guarded by a length check or already ends in an exception, the eleven indexed writes are either on plain arrays or in bounds, unset() on a buffer never happens, no toInt() sees more than four bytes, no concat() passes a total length, and Buffer's own read*()/write*() accessors are never called - the readUInt8() here is BinaryParser's.
One case did change behaviour, and the hole predates the upgrade: Ctid never validated the length of its input. Ctid::fromCtid('C002') threw a ValueError out of slice() under 1.x and would now answer getTransactionIndex() with 0. The constructor requires the sixteen hexadecimal characters a CTID has.
Several places concatenated buffers by unpacking them into int arrays, array_merge()-ing those and rebuilding a buffer - work Buffer::concat() does directly. BytesList::toBytes() is the one that matters, since every serialized field passes through it and it merged in a loop. Amount::fromJson(), AddressCodec::encodeXAddress(), which also went out through sprintf('%02X'), join() and a hex decode, and BinaryParser::readUIntN(), which unpacked and rebuilt the buffer it had just read, do the same on a smaller scale. Encodings are unchanged, byte for byte.
Buffer 2.0 types toArray() as list<int>, which made a cast in BaseX::encode() redundant and Psalm fail the run; the cast is gone.
3000 iterations over an IOU Payment, median of three runs, for buffer 1.0.1, buffer 2.0 and 2.0 with the concat changes: encode 1005 -> 797 -> 780 ms, decode 1364 -> 908 -> 818 ms, encodeXAddress 305 -> 304 -> 280 ms. The string backing of 2.0 accounts for most of it.
3.0.0 stays reserved for removing the Xahau types once hardcastle/xahau_php exists, so this goes out as a minor. The one thing consumers have to know is in the entry: code of theirs that reads ->length on a Buffer this library handed it needs getLength(). That method has existed since Buffer 1.0, so the change is valid under both majors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two pieces of work, both of them in the binary codec, released together as 2.4.0.
The nested
ObjectEndMarkerStObject::fromJson()wrote theObjectEndMarkeronce, after its loop, guardedby whichever field instance the loop happened to leave behind. Fields are
serialized in canonical order, so a nested object is the last field of its parent
only as long as no field of a higher numbered type follows it — and where one
does, the marker was never written and the parser reads that following field back
as part of the nested object.
Transaction metadata is exactly that case on the XRP Ledger: every
ModifiedNodecarrying both
PreviousFieldsandFinalFieldsencoded the second inside thefirst.
fromParser()already wrote the marker per field;fromJson()now doesthe same. Encodings that were correct before are unchanged, byte for byte.
Covered by
testEncodeDecodeObjectFollowedByAnotherFieldandtestEncodeDecodeAdjacentObjects.hardcastle/buffer2.0The constraint moves to
^2.0. 2.0 makesBuffer::$lengthprivate and ships sixbehaviour fixes; each was checked against this codebase and none is reachable:
$lengthno longer publicslice()past the end returns empty instead of throwing$buf[$i] = $xpast the end is a no-opunset($buf[$i])is a no-optoInt()throws beyond 8 bytesconcat($list, $totalLength)zero-fillsread*()/write*()throwreadUInt8()here isBinaryParser'sOne case did change behaviour, and it turned out to be a hole that predates the
upgrade:
Ctidnever validated the length of its input.Ctid::fromCtid('C002')threw a
ValueErrorout ofslice()under 1.x and would now answergetTransactionIndex()with0. The constructor requires the sixteenhexadecimal characters a CTID has, with a regression test.
Consumers reading
->lengthon aBufferthis library handed them needgetLength(), which has existed since Buffer 1.0 and works under both majors.While in there
Several places concatenated buffers by unpacking them into
intarrays,array_merge()-ing those and rebuilding a buffer — workBuffer::concat()doesdirectly.
BytesList::toBytes()is the one that matters, since every serializedfield passes through it and it merged in a loop.
Amount::fromJson(),AddressCodec::encodeXAddress()(which also went out throughsprintf('%02X'),join()and a hex decode) andBinaryParser::readUIntN()(which unpacked andrebuilt the buffer it had just read) do the same thing on a smaller scale.
Encodings unchanged.
Buffer 2.0 types
toArray()aslist<int>, which made a cast inBaseX::encode()redundant and Psalm fail the run; the cast is gone.Verification
610 unit tests green,
composer psalmexits 0.3000 iterations over an IOU
Payment, median of three runs:concatencodedecodeencodeXAddressThe string backing of Buffer 2.0 accounts for most of that; the
concatchangesadd roughly 10% on top for decoding.