Add a parser - #5
Open
pjhartzell wants to merge 28 commits into
Open
Conversation
- Add parser/ Go module at github.com/Element84/metolia-spec/parser - Add go.work including only ./parser, keeping Hugo's root module out - Define the catalog.Resolver interface: provider URI to definition bytes
- Add parser/DESIGN.md capturing the design decisions so far: monorepo workspace, open/closed-source posture, catalog-free core parser, provider-aware validation as a separate future layer, embedded schemas - Remove parser/catalog/resolver.go: with the core parser catalog-free, provider resolution is no longer a parser concern and moves to the future validation layer
- Add decisions 6-13: Step as a closed interface sum, positioned-tree decoding, document-order collections, expression recognition without CEL, hand-written structural validation in the bind pass, collected diagnostics with severities, by-product-only warnings, and a bytes-not-files API surface - Revise decision 5: published schemas become a differential-test oracle, replacing the embedded-copy drift check
- decode.go: token-level decode into a positioned value tree with exact token offsets, duplicate-member rejection, and trailing-content and syntax-error diagnostics - value.go: Value/Member tree nodes, members in document order - diagnostic.go: Severity, Position, Diagnostic, and Diagnostics types - expr.go: ExprBody whole-string expression classification - doc.go: package documentation, including the linter boundary - tests: unit coverage plus a smoke test over the schema fixture corpus
- Add "The parser is three passes" entry with the decode/bind/check table as the pipeline's home - Merge the structural-validation and test-oracle entries into one - Fix stale reference to the dropped embedded-schema drift check - Unify terminology on "pass" and regroup the glance list by topic - Record that positions are stored fully resolved on tree nodes
- ast.go: Flow, the seven Step variants as a closed interface sum, calls, middleware, catch clauses, and the expression-bearing field types (Expr, OrExpr, RetryableField, PreviousField, ConcurrencyCap) - bind.go: the structural pass from value tree to typed AST; validates shapes, required and unknown members, exclusive-member rules, provider URIs including dot-segment rejection, and structural-field expression rejection - diagnostic.go: enumerate all error codes as documented constants - decode.go: use the code constants - value.go: document Member's two roles (tree object member, ordered name-to-value map in the AST) - bind_test.go: fixture-corpus agreement, coverage-fixture spot checks, and 43 targeted diagnostic cases
- check.go: resolve entrypoint and next in their own steps map and flow names along the lexical scope chain, detect flow-reference cycles by depth-first search, warn on shadowed flows entries - check_test.go: resolution, cycle, and shadowing cases, plus the valid corpus through all three passes - diagnostic.go: add the check-pass error codes, the warning-code list, and severityOf as the single source of each code's severity - decode.go, bind.go: emit severity via severityOf
- parser.go: ParseFlow runs decode, bind, and check, sorts diagnostics into document order, and returns a nil Flow on any error; Option and WithFilename supply the diagnostic filename label - parser_test.go: nil-on-error at each pass, warnings preserving the Flow, cross-pass diagnostic ordering, filename labeling, and the fixture corpus through the public API
…loor - Replace deprecated languageCode key, fatal on Hugo 0.164+ - Declare module.hugoVersion min 0.158.0 extended, the floor the locale key requires, so an unsupported Hugo fails with a clear version message
- cmd/mwl: new zero-dependency module in the workspace; binary
named by its directory per go install convention
- main.go: parse flow subcommand with --ast and --strict, "-" for
stdin, hand-rolled flag scan so flags may follow the positional
- Streams: text diagnostics on stderr always; stdout carries only
the --ast envelope {filename, ast, diagnostics}
- Exit codes: 0 parsed, 1 rejected (warnings too under --strict),
2 tool failure
- astjson.go: hand-designed AST encoding with ordered keys, tagged
literal-or-expression arms, classified template string leaves,
number literals as source text, pos last
- Golden tests: byte-exact fixture-to-output pairs for both
streams; regenerate with go test -update
- cmd/mwl/DESIGN.md: decision record for the CLI surface
- dprint.json: exclude cmd/mwl/testdata, goldens bake fixture
byte offsets
- CLAUDE.md, README.md: document the Go workspace modules and
golden workflow
- Replace per-type encoder functions with a reflective walker; golden output is byte-identical - Hand-written code now covers only what struct shapes cannot express: action discriminator, positions, value trees with classified leaves, explicit-null pair types, key rules - A new AST field appears in output with no encoder change; a new Step variant fails tests until its discriminator is mapped - DESIGN.md: record the derivation approach and the field-rename coupling it introduces
- State the boundary: envelope shapes mirror the AST types for tooling, with no readability accommodations; tried-and-removed half-measures noted - Mark MWL re-emission as the human-facing counterpart and first in line among the deferrals
- Add decision 10: bind types every construct the spec constrains and retains a positioned value tree where its opinions stop (a flow's template fields, a provider's embedded schemas) - Renumber the decisions-at-a-glance list accordingly
…der's prefix - providers/_index.md: state in the failure-catalog section that closed codes and non-`*` open sub-prefixes sit under the provider's own URI kind and codePrefix; validators MAY warn on a mismatch, MUST NOT reject - conformance.md: add the matching tooling row to the Providers requirements index Schemas and fixtures unchanged: the finding is advisory, so no document's accept/reject verdict moves.
- No third-party packages and no stdlib flag: dispatch is hand-written either way, flag stops at the first non-flag argument (breaking flags-after-file), and the exit-2 contract stays on one explicit path - Cobra threshold unchanged
- Rework decision 7's closing paragraph: a union is arms on its shared core, graduating to a closed sum when the variant count makes the one-arm invariant illegible and exhaustiveness needs a machine check - Record the provider definition's two kinds as an arms shape: one Provider struct, shared fields flat, a pointer arm per kind
- middleware-providers.md: state in the structural-parameters section that a structural name matching no key of the phase's parameters top-level properties is legal but silently dead; validators MAY warn, MUST NOT reject - conformance.md: add the matching tooling row to the Providers requirements index Schemas and fixtures unchanged: the finding is advisory, so no document's accept/reject verdict moves.
- valid/provider/: two coverage fixtures for the branches the five reference definitions leave unused (no-metadata call provider, long URI with full segment charset, multi-code catalog, non-wildcard open sub-prefix; single-level attachment, all four phases, transform action, parameters plus structural in one phase) - invalid/provider/: eighteen new fixtures, one per unexercised schema rule, each a minimal document failing for exactly its named reason
parser: - Add ParseProvider: bind and check passes over the shared decode - One Provider struct: shared fields flat, a pointer arm per kind, kind discriminated by the URI's type segment - Retain embedded parameter and metadata schemas as value trees - Warn on catalog entries outside the provider's kind/codePrefix and on structural names matching no parameters property - Publish action kinds, attachment levels, Result types as constants - Bind an explicit concurrency null as absent; ConcurrencyCap carries a plain value - Split files into a shared/flow/provider grid; move the flow graph onto a flowChecker - Pin per-rule diagnostics in provider bind/check tables; run the provider corpus through ParseProvider cmd/mwl: - Wire mwl parse provider (--ast, --strict) via a shared runner - Encode Provider through the reflection walker; drop encConcurrency - Reorganize testdata into flow/ and provider/; add provider goldens docs: - Record decisions 15 (published constants) and 16 (empty-spelling normalization); sharpen decision 8's round-trip wording - Rephrase narrated-history wording in both DESIGN documents
- Rewrite decision 11's test paragraphs: agreement between parser and schemas is held transitively through the shared fixture corpus by two suites (check-schemas.sh schema-side, the corpus tests parser-side), with no JSON Schema engine and a zero-dependency go.mod even for tests - Record the accepted limit: agreement is only as wide as the corpus
- Add a per-segment negative lookahead to the call and middleware URI patterns in both schemas: the prose forbids "." and ".." as URI segments, but the charset alone admitted them (the parser already rejected them) - Legal dot-bearing segments (v1.2, .hidden, "...") still match - Add dot-segment invalid fixtures on both the flow and provider sides, encoding the divergence in the corpus
- ast_flow.go: replace block field comments with go/ast-style trailing comments, one clause per field; promote invariants and non-obvious semantics to the type doc comments - ast_provider.go: same conversion for the provider AST - ast.go: comment Expr.Body and the arms of the three-way fields - diagnostic.go: convert Diagnostic.Code and .Path to trailing comments
- README.md: the decode/bind/check pipeline, the parse contract, a file map, and the deliberate boundaries; DESIGN.md keeps the why - doc.go: name the three passes in the package comment; say "checks" rather than "findings" for analyses beyond the parser's scope
…er pages - content/tools/: section landing page, parser overview, mwl CLI page, and the AST explorer — an interactive field-by-field reference for the Flow and Provider trees the parser returns - scripts/check-explorer.py: drift check diffing the explorer page's data-field/data-type attributes against the parser AST declarations - lefthook.yml: run the explorer check pre-commit on staged go/md/py - scripts/check-build.sh: widen the HTML anchor check to cover the tools section alongside the reference - hugo.yaml, content/_index.md: add the Tooling menu entry and landing card between Reference and Rationale - layouts/_default/list.specmarkdown.md: skip sections flagged singleFileExclude in the whole-site spec.md; content/tools sets it - dprint.json: exclude ast-explorer.md (raw interactive HTML rows) - CLAUDE.md: document the section, the drift check, the SpecMarkdown exclusion, and the Python 3.10 floor for check-explorer.py
- Publish the action discriminator as Step.Action(), one method per variant; drop the CLI's type-to-action table and its panic path - Extract bindResultType for the non-success type check both constructed failures share - Share the invalid-provider-URI message via a binder helper - Unify the per-pass diagnostic accumulators into one reporter in diagnostic.go; delete check.go and the decoder's bespoke copy - Key the AST encoder's policy tables by (type, field) instead of bare field name; keep Comment as an AST-wide convention beside Pos - Amend both DESIGN.md files and the parser README accordingly
- Standardize generic tool references on "the MWL CLI"; reserve backticked mwl for literal command and binary contexts - Record the casing rule in CLAUDE.md's Casing section - Set cmd/mwl/README.md's heading in code font - Align the home page's Tooling card subtitle
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a parser for the MWL language. A CLI and extended documentation in the
hugowebsite are also included in this PR.The parser is the key feature. It operates on an MWL document in three passes, each consuming the previous pass's output, with a single diagnostic list that all passes append to.
decode — turns the source bytes into a value tree: the whole document held in a single struct,
Value, which represents any JSON value (null, boolean, number, string, object, array) and nests instances of itself — an object holds its members as name/Valuepairs, an array holds its elements asValues — so the document forms a tree whose nodes areValues. Every value and every object key is tagged with its byte offset, line, and column, and object members are kept as an ordered slice rather than a map, so source order survives and duplicate keys are detectable. Each node is labeled only by its JSON type; no MWL meaning is attached yet. This pass rejects malformed JSON, empty documents, duplicate object members, and trailing content. No later pass touches the bytes.bind — walks the value tree and builds the typed AST, whose nodes are the language's own shapes (
Flow,Step,Call); this is where MWL meaning first exists. Shape is enforced during the translation itself, so no AST node is ever built from a malformed value and nothing downstream re-checks: each value must be the JSON type its field requires (nextmust be a string,stepsan object), objects may carry only the members their shape accepts (and must carry the required ones), exactly-one pairs (a call'sproviderorflow, a Sleep'sfororuntil) must have exactly one side present, and expressions may appear only where permitted. A violation becomes a diagnostic and binding continues, so one parse reports every shape problem in the document.check — runs the rules that span the bound document, which no single field's shape can express. Every
entrypointandnextmust resolve to a step; a call's flow name must resolve to aflowsdeclaration, looked up innermost-outward through the enclosing Flows the way variable lookup walks nested scopes; and the flow-reference graph must be acyclic. It also produces all of the parser's warnings, where a document is legal but rarely what the author meant: aflowsentry that shadows an outer declaration of the same name, a failure-catalog entry whose code sits outside the provider's own kind and prefix, astructuraldeclaration naming no parameter of its phase (so it marks nothing). It runs only on an AST that bound without errors.See also the
README.mddocuments in theparserandcmd/mwlmodules. Launch thehugosite locally to view an AST explorer.