refactor(json): scan string escapes with lexscan - #4095
Open
bobzhang wants to merge 1 commit into
Open
Conversation
bobzhang
commented
Aug 18, 2026
bobzhang
commented
Contributor
Step 2 of the plan behind #4094: with the parser state now being @lexbuf.StringScanner, `lexscan` can drive parts of the lexer directly. This converts the designated performance-non-critical spot — the string slow path, entered only when a string contains an escape — to a lexscan loop whose cases read like RFC 8259 §7: closing quote, longest ordinary run, the eight single-character escapes, a BMP `\uXXXX` (first-digit classes exclude the surrogate range), and a whole escaped surrogate pair in one regex. The manual run bookkeeping (`start`/`flush`) is gone; the compiler statically rejects shadowed cases (unused_lexcase). Everything invalid falls to the wildcard, which leaves the cursor at the failure point; `lex_string_failure` re-reads it with the old per-character primitives so every error position is unchanged: bad escape characters and hex digits are blamed themselves, unpaired surrogate escapes at their opening backslash, EOF raises InvalidEof. The 209-test suite (including the parse_error suites pinning exact positions for every malformed-escape shape) passes unchanged on wasm-gc, native and js, plus a new test for astral characters inside slow-path runs — the negated class traverses surrogate pairs, and raw lone surrogates still pass through as before (probed). Measured on a new escape-heavy bench (previously zero bench coverage for this path): js 687->682us and wasm-gc 1.21->1.22ms are unchanged; native is 473->840us (~1.8x slower on escape-heavy strings), the cost of DFA-generated code on the RC backend. Accepted deliberately: this path is cold (ordinary strings never enter it), and the readability is the point of the experiment. Fast-path benches are unchanged on all targets. Codex CLI review: approved — "Regex coverage, fallback error positions, hex classes, surrogate arithmetic, and comments are consistent with the prior behavior and RFC 8259. The hex partition and full surrogate range also check out exhaustively." Signed-off-by: Codex CLI <codex@openai.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.