fix: Make the JSON tokenizer and writer RFC 8259 compliant - #48
Merged
Conversation
Port of the v3 RFC 8259 compliance work (#46) to the v4 module line. The v4 tokenizer/writer code is identical to v3's aside from the (already removed) easyjson build tags, so this applies the same logic; the tests drop the build tag to match v4's single-backend layout. Reader: - Whitespace: allow only space, tab, LF, and CR between tokens. - Numbers: reject leading zeros, a bare minus, and a trailing decimal point; integers that overflow int64 fall back to float parsing so they match encoding/json instead of returning a wrapped value. - Strings: reject unescaped control characters (below 0x20); substitute the Unicode replacement character for invalid UTF-8 bytes; combine UTF-16 surrogate pairs into a single code point (lone/invalid surrogates decode to U+FFFD), matching encoding/json. Writer: - Float64 returns an error for NaN and infinities rather than emitting invalid JSON.
keelerm84
marked this pull request as ready for review
July 29, 2026 17:37
kinyoklion
approved these changes
Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of the v3 RFC 8259 compliance work (#46) to the v4 module line. The v4
tokenizer/writer code is identical to v3's aside from the (already removed)
easyjson build tags, so this applies the same logic; the tests drop the build
tag to match v4's single-backend layout.
Reader:
integers that overflow int64 fall back to float parsing so they match
encoding/json instead of returning a wrapped value.
Unicode replacement character for invalid UTF-8 bytes; combine UTF-16
surrogate pairs into a single code point (lone/invalid surrogates decode to
U+FFFD), matching encoding/json.
Writer:
JSON.
Note
Medium Risk
Changes core JSON parsing/encoding semantics and will reject or decode some inputs differently than before; behavior is heavily tested against encoding/json but any caller relying on the old lax behavior could break.
Overview
Brings the v4 jreader / jwriter default tokenizer in line with RFC 8259 and
encoding/jsonbehavior (port of prior v3 work).Reader: Token whitespace is limited to space, tab, LF, and CR (no other Unicode spaces). Number parsing is rewritten to enforce JSON grammar (leading zeros, incomplete fractions/exponents, etc.); int64 overflow no longer wraps—literals fall through to
strconv.ParseFloatlike the stdlib. Strings reject unescaped control chars; invalid UTF-8 and lone/malformed\usurrogates decode likeencoding/json(replacement char, surrogate-pair combining via newreadUnicodeEscape).Writer:
Float64now errors on NaN and ±Inf instead of emitting invalid JSON.New RFC-focused test suites compare accept/reject and decoded values against
encoding/json.Reviewed by Cursor Bugbot for commit 0cd02e8. Bugbot is set up for automated code reviews on this repo. Configure here.