refactor(json): parse on @lexbuf.StringScanner instead of a private ParseContext - #4094
Open
bobzhang wants to merge 1 commit into
Open
refactor(json): parse on @lexbuf.StringScanner instead of a private ParseContext#4094bobzhang wants to merge 1 commit into
bobzhang wants to merge 1 commit into
Conversation
bobzhang
commented
Aug 18, 2026
bobzhang
commented
Contributor
…arseContext The json parser's state was a private struct with the same shape as @lexbuf.StringScanner, the non-streaming scanner target of `lexscan` (data + cursor over a StringView, both in UTF-16 code units). Delete it and define the lexer/parser methods directly on StringScanner as local methods, so a scanner mid-parse is exactly the value `lexscan` consumes — groundwork for moving parts of the lexer to `lexscan` in a follow-up PR. The only non-rename change is dropping the cached `end_offset` field: every former read is `data.length()` now, an O(1) intrinsic (end - start). Benches: native slightly faster than main (mixed 1.12ms vs 1.16ms, int 500us vs 528us, float 725us vs 770us); js statistically unchanged (3-run A/B on the int-array bench, ~969us vs ~970us). Fully qualified `@lexbuf.StringScanner` is written throughout because the current nightly deprecates `typealias` / `using ... as` renames and warns (unqualified_local_using) on the replacement form, and CI runs `moon check --deny-warn`. Codex CLI review: approved — "All eight former ctx.end_offset reads map to ctx.data.length() with their bounds unchanged. The offset/input migration is complete, public interfaces remain unaffected, comments are accurate." Signed-off-by: Codex CLI <codex@openai.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Coverage Report for CI Build 6145Coverage increased (+0.001%) to 90.877%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
Contributor
There was a problem hiding this comment.
Pull request overview
Refactors the JSON parser/lexer to use the shared non-streaming scanner @lexbuf.StringScanner (from lexbuf) instead of a private ParseContext, aligning JSON parsing with the common scanning infrastructure.
Changes:
- Replaces
ParseContextwith@lexbuf.StringScanneracross parsing and lexing code paths (offset/input/end_offset→cursor/data/length()). - Removes the private
ParseContexttype and updates error reporting helpers to operate onStringScanner. - Adds a dependency on
moonbitlang/core/lexbuffor the JSON package.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| json/utils.mbt | Moves invalid_char helper onto @lexbuf.StringScanner and updates position/character lookup to use data/cursor. |
| json/parse.mbt | Initializes parsing state with @lexbuf.StringScanner and ports parse entrypoints to operate on it. |
| json/moon.pkg | Adds the lexbuf package import required for @lexbuf.StringScanner. |
| json/lex_string.mbt | Ports string lexing logic to use StringScanner (cursor/data) while preserving surrogate/escape handling. |
| json/lex_number.mbt | Ports number scanning/lexing logic to StringScanner and updates all view/unsafe_get reads to data. |
| json/lex_misc.mbt | Ports low-level scanning helpers (read_char, expect_*, whitespace skipping) to StringScanner. |
| json/lex_main.mbt | Ports main token lexing entry (lex_value) to StringScanner, updating number start offsets to use cursor. |
| json/internal_types.mbt | Removes the private ParseContext definition and documents the new scanner-based state model. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
65
to
+69
| guard c1 is (SURROGATE_LOW_CHAR..=SURROGATE_HIGH_CHAR) && | ||
| ctx.offset < ctx.end_offset else { | ||
| ctx.cursor < ctx.data.length() else { | ||
| ctx.invalid_char(shift=-1) | ||
| } | ||
| let c2 = ctx.input.unsafe_get(ctx.offset).to_int() | ||
| let c2 = ctx.data.unsafe_get(ctx.cursor).to_int() |
Comment on lines
+15
to
23
| // The parser's state is `@lexbuf.StringScanner`, the non-streaming scanner | ||
| // target shared with `lexscan`, rather than a private equivalent: `data` is | ||
| // the input view and `cursor` is the relative offset of the next code unit, | ||
| // both in UTF-16 code units. The lexer methods on it live in this package; | ||
| // a scanner mid-parse can therefore be handed to `lexscan` directly if parts | ||
| // of the lexer move there later. | ||
|
|
||
| ///| | ||
| priv enum Token { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.