Skip to content

fix: Make the JSON tokenizer and writer RFC 8259 compliant - #48

Merged
keelerm84 merged 1 commit into
v4from
mk/SDK-2771/rfc-8259-compliance-v4
Jul 29, 2026
Merged

fix: Make the JSON tokenizer and writer RFC 8259 compliant#48
keelerm84 merged 1 commit into
v4from
mk/SDK-2771/rfc-8259-compliance-v4

Conversation

@keelerm84

@keelerm84 keelerm84 commented Jul 29, 2026

Copy link
Copy Markdown
Member

Port of the v3 RFC 8259 compliance work (#46) to the v4 module line. The v4
tokenizer/writer code is identical to v3's aside from the (already removed)
easyjson build tags, so this applies the same logic; the tests drop the build
tag to match v4's single-backend layout.

Reader:

  • Whitespace: allow only space, tab, LF, and CR between tokens.
  • Numbers: reject leading zeros, a bare minus, and a trailing decimal point;
    integers that overflow int64 fall back to float parsing so they match
    encoding/json instead of returning a wrapped value.
  • Strings: reject unescaped control characters (below 0x20); substitute the
    Unicode replacement character for invalid UTF-8 bytes; combine UTF-16
    surrogate pairs into a single code point (lone/invalid surrogates decode to
    U+FFFD), matching encoding/json.

Writer:

  • Float64 returns an error for NaN and infinities rather than emitting invalid
    JSON.

Note

Medium Risk
Changes core JSON parsing/encoding semantics and will reject or decode some inputs differently than before; behavior is heavily tested against encoding/json but any caller relying on the old lax behavior could break.

Overview
Brings the v4 jreader / jwriter default tokenizer in line with RFC 8259 and encoding/json behavior (port of prior v3 work).

Reader: Token whitespace is limited to space, tab, LF, and CR (no other Unicode spaces). Number parsing is rewritten to enforce JSON grammar (leading zeros, incomplete fractions/exponents, etc.); int64 overflow no longer wraps—literals fall through to strconv.ParseFloat like the stdlib. Strings reject unescaped control chars; invalid UTF-8 and lone/malformed \u surrogates decode like encoding/json (replacement char, surrogate-pair combining via new readUnicodeEscape).

Writer: Float64 now errors on NaN and ±Inf instead of emitting invalid JSON.

New RFC-focused test suites compare accept/reject and decoded values against encoding/json.

Reviewed by Cursor Bugbot for commit 0cd02e8. Bugbot is set up for automated code reviews on this repo. Configure here.

Port of the v3 RFC 8259 compliance work (#46) to the v4 module line. The v4
tokenizer/writer code is identical to v3's aside from the (already removed)
easyjson build tags, so this applies the same logic; the tests drop the build
tag to match v4's single-backend layout.

Reader:
- Whitespace: allow only space, tab, LF, and CR between tokens.
- Numbers: reject leading zeros, a bare minus, and a trailing decimal point;
  integers that overflow int64 fall back to float parsing so they match
  encoding/json instead of returning a wrapped value.
- Strings: reject unescaped control characters (below 0x20); substitute the
  Unicode replacement character for invalid UTF-8 bytes; combine UTF-16
  surrogate pairs into a single code point (lone/invalid surrogates decode to
  U+FFFD), matching encoding/json.

Writer:
- Float64 returns an error for NaN and infinities rather than emitting invalid
  JSON.
@keelerm84
keelerm84 marked this pull request as ready for review July 29, 2026 17:37
@keelerm84
keelerm84 requested a review from a team as a code owner July 29, 2026 17:37
@keelerm84
keelerm84 merged commit 6efdc00 into v4 Jul 29, 2026
14 checks passed
@keelerm84
keelerm84 deleted the mk/SDK-2771/rfc-8259-compliance-v4 branch July 29, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants