Update to mojo 1.1 - #349
Closed
lsh wants to merge 80 commits into
Closed
Conversation
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Add traits support
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <1734032+lsh@users.noreply.github.com>
Add mut/read keywords and pull from main
Signed-off-by: Lukas Hermann <1734032+lsh@users.noreply.github.com>
add out keyword
Signed-off-by: Lukas Hermann <lukashermann28@gmail.com>
Signed-off-by: Lukas Hermann <1734032+lsh@users.noreply.github.com>
Allow each entry of a struct's superclass list to carry `where` constraints, e.g. `struct Tuple[...](Copyable where AllCopyable[*element_types], ...)`. A dedicated rule aliased to `argument_list` keeps the node type unchanged.
Accept `name = start:stop` keyword arguments inside a subscript, e.g. `buf[byte=1:end]` or `s[byte=:stop]`, via a slice-valued keyword argument scoped to subscripts.
Accept `def f() raises HALError:` where a typed `raises` is followed directly by the function body colon. The error type is modelled so it never forms a top-level `constrained_type`, whose `:` would otherwise be read as the body colon, while parametric/unioned/dotted error types still parse as proper types.
Allow the argument-convention soft keywords to be used as ordinary names or values: `mut`/`out` via `keyword_identifier` (e.g. `mut == False`), and `read` by leaving it out of the reserved-word set so it lexes as a plain identifier where a convention is not expected (e.g. a method `def read(self)` or `UInt(read)`) while still being keyword-extracted for `read x` conventions.
Allow each convention in a brace-delimited convention clause to bind a name,
e.g. `def f() {read x, mut y}:`, while keeping the bare `{read}` form.
Accept a compile-time call used as a type argument, e.g. `Device[get_device_spec[0]()]`, where the generic-type reading would otherwise strand the trailing `()`. Also handles a dotted parametric base (`TypeList.splat[...]()`), a trailing member-call chain (`get_device_spec[0]()._mlir_target()`), and splatting such types (`*Self.element_types.reverse()`).
Accept a backtick-quoted (raw) identifier as a binding name, e.g. ``var `6bit` = ...``. Such names lex as a (string), so this admits a string as an assignment/loop target alongside the usual patterns.
Allow a call (or method-call chain) on the left-hand side of an assignment, e.g. `self.get(i) = x` or `node[].right() = other`.
Add the `__extension Type:` compound statement, e.g. `__extension List:` or `__extension SIMD[T, n]:`.
Allow a callable type's value parameters to carry an argument convention, e.g. `def(mut Bencher, T) raises capturing[_] -> None`.
Accept a relative import that uses `import` directly (a leading dot), e.g. `import .warp`, alongside the existing `from .mod import ...` form.
Accept an argument convention on a `for` loop variable, e.g. `for var arg in argv:` or `for ref item in items:`.
Accept an `abi("C")` qualifier on a function definition or callable type, e.g.
`def f() abi("C"):` or `def(PyObjectPtr) thin abi("C") -> PyObjectPtr`.
Accept `comptime (expr)` in value position, e.g. `result[i] = comptime (StaticString(raw[i]))`.
Accept the `&` intersection/conjunction operator combining trait/types with a callable type, e.g. a parameter bound `Copyable & RegisterPassable & def(Int) -> Int`. A trailing `function_type` is required so a plain `A & B` of expressions still parses as a `binary_operator`.
Allow a decorator to be applied to a `comptime` alias, e.g. `@deprecated(...)` preceding `comptime X = Y`.
Accept more than one argument in a `ref[...]` origin convention, e.g. `ref[origin, address_space] self`.
The MLIR interop change lexed backtick fragments as plain strings, so a highlighter could not distinguish ``__mlir_op.`pop.cast` `` from a string literal. Introduce a dedicated `mlir_fragment` token for the dotted backtick forms (`__mlir_op.`op``, `__mlir_type.`type``, `__mlir_attr.`attr``) and highlight it as a type. Replacing the over-general `obj.<string>` attribute branch with the backtick-only fragment also restores the expected error recovery in the "error before a string literal" test (now passing).
Previously a backtick MLIR fragment was a single opaque token, so its interior highlighted as one flat span (a regression from the older structured parse). Add an external-scanner mode that, inside a backtick MLIR fragment, lexes the interior into typed identifiers, numbers and punctuation while suppressing the comment and string lexers between the backticks. This restores per-element highlighting (types/literals/operators) and, unlike the old structured grammar, stays robust on `#`-prefixed attributes and `"`-containing fragments such as `#kgen.dtype.constant<ui8> : !kgen.dtype` (no parse errors). Raw-identifier backticks in expression position are unaffected and still lex as strings.
Fix parse errors in Mojo stdlib
queries/tags.scm was deleted in 0a71ddb ("more tweaks"), but bindings/rust/lib.rs still had pub const TAGS_QUERY: &str = include_str!("../../queries/tags.scm"); so the Rust crate has failed to compile at every rev since: error: couldn't read .../queries/tags.scm: No such file or directory Drop the TAGS_QUERY const and the stale "tags" entries in tree-sitter.json and package.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolve tree-sitter.json conflict by keeping the restored queries/tags.scm reference and the mojo injection-regex from this branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Clean up Mojo project metadata
Rewrite the grammar against the current compiler's parser (Mojo/lib/MojoParser)
and validate it against real code.
Removed: `fn`, `borrowed`/`inout`/`owned`/`read` conventions, `escaping` and
`unified` effects, `del`/`global`/`nonlocal`/`exec`/`print` statements, the
`print >> x` chevron, parameter-list `where` clauses, struct field default
initializers, and `\N{...}` escapes — all hard parse errors now.
Added: closure capture lists on nested defs and lambdas, lambda expressions,
contextually inferred member refs (`.red`), t-strings, `abi(...)`/`thin`
effects, `raises E`, trailing `where` constraints, function-type origin sets,
`__generator_type`, `Self`, `ref[origin]` conventions, and `__match` (the only
match spelling the compiler parses).
Parse sweep over the modular checkout drove out a further round of fixes:
`where` on associated and parameterized comptime aliases, associated aliases
with no value, supertraits as type expressions, callable-type conformances,
extension conformance lists, backtick raw identifiers as keyword-argument
names / definition names / module-path components, decorated imports, variadic
and separated callable-type parameters, parenthesized callable types, callable
operands anywhere in an intersection, indexed parametric call results,
arithmetic and parametric members in parameter positions, `var`/`ref` bindings
in patterns and call arguments, walrus on member/index/tuple targets, and the
`__param_trait__` effect.
Queries: highlights.scm audited against node-types.json — added type rules for
class/trait/extension/alias names, parameter and keyword-argument rules (placed
before the `^[A-Z]` heuristic so `def f[T: AnyType]` keeps `T` as a type),
struct-literal fields, `self`, and `__param_trait__`. outline.scm and tags.scm
gained `extension_definition`.
Tests: 221 corpus tests pass; highlight fixtures cover keywords, parameters,
pattern matching and definitions; tags fixture extended.
Real-world coverage: mojo/stdlib/std parses 249/249 (100%); the whole modular
checkout 7063/7352, where the remainder is dominated by the compiler's own
parser error tests and black's Python test data.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drop the external scanner's backtick-fragment tokenizer (MLIR_BACKTICK, MLIR_IDENT, MLIR_NUMBER, MLIR_PUNCTUATION), the `mlir_fragment` and `mlir_type` rules, the `mlir_punctuation` node, and the `mlir_*` highlight queries. MLIR interop now parses as ordinary expressions: `__mlir_op`, `__mlir_attr` and `__mlir_type` are plain identifiers, and a backtick-quoted member (``__mlir_op.`pop.cast` ``) is an `attribute` whose name is a (string) — the same shape backticks already had as binding names (``var `6bit` = ...``). `__mlir_region` is kept: it is a plain statement rule, not part of the op-specific handling. Coverage is unchanged: mojo/stdlib/std still parses 249/249, and the whole modular checkout 7063/7352. One cost: without a scanner token the grammar cannot tell a backtick string from a quoted one, so (string) in member position also admits `c."..."`. The errors.txt recovery case that relied on `c.` followed by a docstring being an error now uses `c.-` to test the same thing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`type` re-derived the whole expression grammar alongside a dozen type-only
rules, so every type position duplicated expression states. That cost a bloated
parse table and blocked `intersection_type` in value position: adding it to
`_right_hand_side` made `generic_type` reachable there, hijacking ordinary
right-hand sides (`x = Foo[]` became a parametric type, `ref r = a[1]` broke),
because the reduce/reduce between `type -> expression` and `type -> generic_type`
was settled by static precedence with no GLR fork available to defer the choice.
Mojo spells types with the expression grammar, so `type` is now that plus the
three forms with no expression spelling:
type: choice($.expression, $.intersection_type, $.function_type,
$.generator_type)
`generic_type`, `called_type`, `member_type`, `union_type` and `splat_type` are
gone as nodes — a parametric instantiation is a `subscript` (`List[Int]`), a
qualified name an `attribute` (`Self.T`), a specialization-and-call a `call`
(`get_device_spec[0]()`), a union a `binary_operator` (`Int | None`), a variadic
a `list_splat` (`*Ts`). The bespoke `type_arithmetic` and
`parametric_member_type` rules went with them: `size_of[T]() * 2` and
`CompTest[.int32].Sq[10]` are ordinary expressions now. `intersection_type`
shares the expression chain too — `seq($.primary_expression, '&', callable)` —
so `A & B & C` stays a `binary_operator` until a callable operand turns up.
This is a breaking change for tree consumers: queries matching
generic_type/called_type/member_type/union_type/splat_type must match
subscript/call/attribute/binary_operator/list_splat instead. No query in this
repo referenced them.
STATE_COUNT 33362 -> 27966
src/parser.c 49 MB -> 37 MB
generate ~3.5 min -> ~2.5 min
declared conflicts 33 -> 30
std parse 249/249 (unchanged)
whole checkout 7063/7352 -> 7066/7350
The three `comptime X = A & B & (def(...) -> None)` failures are fixed; the six
that remain are all invalid Mojo the grammar is right to reject.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.