Skip to content

Add a parser - #5

Open
pjhartzell wants to merge 28 commits into
mainfrom
feat/pjh/parser
Open

Add a parser#5
pjhartzell wants to merge 28 commits into
mainfrom
feat/pjh/parser

Conversation

@pjhartzell

@pjhartzell pjhartzell commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Adds a parser for the MWL language. A CLI and extended documentation in the hugo website are also included in this PR.

The parser is the key feature. It operates on an MWL document in three passes, each consuming the previous pass's output, with a single diagnostic list that all passes append to.

  1. decode — turns the source bytes into a value tree: the whole document held in a single struct, Value, which represents any JSON value (null, boolean, number, string, object, array) and nests instances of itself — an object holds its members as name/Value pairs, an array holds its elements as Values — so the document forms a tree whose nodes are Values. Every value and every object key is tagged with its byte offset, line, and column, and object members are kept as an ordered slice rather than a map, so source order survives and duplicate keys are detectable. Each node is labeled only by its JSON type; no MWL meaning is attached yet. This pass rejects malformed JSON, empty documents, duplicate object members, and trailing content. No later pass touches the bytes.

  2. bind — walks the value tree and builds the typed AST, whose nodes are the language's own shapes (Flow, Step, Call); this is where MWL meaning first exists. Shape is enforced during the translation itself, so no AST node is ever built from a malformed value and nothing downstream re-checks: each value must be the JSON type its field requires (next must be a string, steps an object), objects may carry only the members their shape accepts (and must carry the required ones), exactly-one pairs (a call's provider or flow, a Sleep's for or until) must have exactly one side present, and expressions may appear only where permitted. A violation becomes a diagnostic and binding continues, so one parse reports every shape problem in the document.

  3. check — runs the rules that span the bound document, which no single field's shape can express. Every entrypoint and next must resolve to a step; a call's flow name must resolve to a flows declaration, looked up innermost-outward through the enclosing Flows the way variable lookup walks nested scopes; and the flow-reference graph must be acyclic. It also produces all of the parser's warnings, where a document is legal but rarely what the author meant: a flows entry that shadows an outer declaration of the same name, a failure-catalog entry whose code sits outside the provider's own kind and prefix, a structural declaration naming no parameter of its phase (so it marks nothing). It runs only on an AST that bound without errors.

See also the README.md documents in the parser and cmd/mwl modules. Launch the hugo site locally to view an AST explorer.

- Add parser/ Go module at github.com/Element84/metolia-spec/parser
- Add go.work including only ./parser, keeping Hugo's root module out
- Define the catalog.Resolver interface: provider URI to definition bytes
- Add parser/DESIGN.md capturing the design decisions so far: monorepo
  workspace, open/closed-source posture, catalog-free core parser,
  provider-aware validation as a separate future layer, embedded schemas
- Remove parser/catalog/resolver.go: with the core parser catalog-free,
  provider resolution is no longer a parser concern and moves to the
  future validation layer
- Add decisions 6-13: Step as a closed interface sum, positioned-tree
  decoding, document-order collections, expression recognition without
  CEL, hand-written structural validation in the bind pass, collected
  diagnostics with severities, by-product-only warnings, and a
  bytes-not-files API surface
- Revise decision 5: published schemas become a differential-test
  oracle, replacing the embedded-copy drift check
- decode.go: token-level decode into a positioned value tree with
  exact token offsets, duplicate-member rejection, and trailing-content
  and syntax-error diagnostics
- value.go: Value/Member tree nodes, members in document order
- diagnostic.go: Severity, Position, Diagnostic, and Diagnostics types
- expr.go: ExprBody whole-string expression classification
- doc.go: package documentation, including the linter boundary
- tests: unit coverage plus a smoke test over the schema fixture corpus
- Add "The parser is three passes" entry with the decode/bind/check
  table as the pipeline's home
- Merge the structural-validation and test-oracle entries into one
- Fix stale reference to the dropped embedded-schema drift check
- Unify terminology on "pass" and regroup the glance list by topic
- Record that positions are stored fully resolved on tree nodes
- ast.go: Flow, the seven Step variants as a closed interface sum,
  calls, middleware, catch clauses, and the expression-bearing field
  types (Expr, OrExpr, RetryableField, PreviousField, ConcurrencyCap)
- bind.go: the structural pass from value tree to typed AST; validates
  shapes, required and unknown members, exclusive-member rules,
  provider URIs including dot-segment rejection, and structural-field
  expression rejection
- diagnostic.go: enumerate all error codes as documented constants
- decode.go: use the code constants
- value.go: document Member's two roles (tree object member, ordered
  name-to-value map in the AST)
- bind_test.go: fixture-corpus agreement, coverage-fixture spot checks,
  and 43 targeted diagnostic cases
- check.go: resolve entrypoint and next in their own steps map and flow
  names along the lexical scope chain, detect flow-reference cycles by
  depth-first search, warn on shadowed flows entries
- check_test.go: resolution, cycle, and shadowing cases, plus the valid
  corpus through all three passes
- diagnostic.go: add the check-pass error codes, the warning-code list,
  and severityOf as the single source of each code's severity
- decode.go, bind.go: emit severity via severityOf
- parser.go: ParseFlow runs decode, bind, and check, sorts diagnostics
  into document order, and returns a nil Flow on any error; Option and
  WithFilename supply the diagnostic filename label
- parser_test.go: nil-on-error at each pass, warnings preserving the
  Flow, cross-pass diagnostic ordering, filename labeling, and the
  fixture corpus through the public API
…loor

- Replace deprecated languageCode key, fatal on Hugo 0.164+
- Declare module.hugoVersion min 0.158.0 extended, the floor the
  locale key requires, so an unsupported Hugo fails with a clear
  version message
- cmd/mwl: new zero-dependency module in the workspace; binary
  named by its directory per go install convention
- main.go: parse flow subcommand with --ast and --strict, "-" for
  stdin, hand-rolled flag scan so flags may follow the positional
- Streams: text diagnostics on stderr always; stdout carries only
  the --ast envelope {filename, ast, diagnostics}
- Exit codes: 0 parsed, 1 rejected (warnings too under --strict),
  2 tool failure
- astjson.go: hand-designed AST encoding with ordered keys, tagged
  literal-or-expression arms, classified template string leaves,
  number literals as source text, pos last
- Golden tests: byte-exact fixture-to-output pairs for both
  streams; regenerate with go test -update
- cmd/mwl/DESIGN.md: decision record for the CLI surface
- dprint.json: exclude cmd/mwl/testdata, goldens bake fixture
  byte offsets
- CLAUDE.md, README.md: document the Go workspace modules and
  golden workflow
- Replace per-type encoder functions with a reflective walker;
  golden output is byte-identical
- Hand-written code now covers only what struct shapes cannot
  express: action discriminator, positions, value trees with
  classified leaves, explicit-null pair types, key rules
- A new AST field appears in output with no encoder change; a new
  Step variant fails tests until its discriminator is mapped
- DESIGN.md: record the derivation approach and the field-rename
  coupling it introduces
- State the boundary: envelope shapes mirror the AST types for
  tooling, with no readability accommodations; tried-and-removed
  half-measures noted
- Mark MWL re-emission as the human-facing counterpart and first
  in line among the deferrals
- Add decision 10: bind types every construct the spec constrains and
  retains a positioned value tree where its opinions stop (a flow's
  template fields, a provider's embedded schemas)
- Renumber the decisions-at-a-glance list accordingly
…der's prefix

- providers/_index.md: state in the failure-catalog section that closed
  codes and non-`*` open sub-prefixes sit under the provider's own URI
  kind and codePrefix; validators MAY warn on a mismatch, MUST NOT
  reject
- conformance.md: add the matching tooling row to the Providers
  requirements index

Schemas and fixtures unchanged: the finding is advisory, so no
document's accept/reject verdict moves.
- No third-party packages and no stdlib flag: dispatch is hand-written
  either way, flag stops at the first non-flag argument (breaking
  flags-after-file), and the exit-2 contract stays on one explicit path
- Cobra threshold unchanged
- Rework decision 7's closing paragraph: a union is arms on its shared
  core, graduating to a closed sum when the variant count makes the
  one-arm invariant illegible and exhaustiveness needs a machine check
- Record the provider definition's two kinds as an arms shape: one
  Provider struct, shared fields flat, a pointer arm per kind
- middleware-providers.md: state in the structural-parameters section
  that a structural name matching no key of the phase's parameters
  top-level properties is legal but silently dead; validators MAY warn,
  MUST NOT reject
- conformance.md: add the matching tooling row to the Providers
  requirements index

Schemas and fixtures unchanged: the finding is advisory, so no
document's accept/reject verdict moves.
- valid/provider/: two coverage fixtures for the branches the five
  reference definitions leave unused (no-metadata call provider, long
  URI with full segment charset, multi-code catalog, non-wildcard open
  sub-prefix; single-level attachment, all four phases, transform
  action, parameters plus structural in one phase)
- invalid/provider/: eighteen new fixtures, one per unexercised schema
  rule, each a minimal document failing for exactly its named reason
parser:
- Add ParseProvider: bind and check passes over the shared decode
- One Provider struct: shared fields flat, a pointer arm per kind,
  kind discriminated by the URI's type segment
- Retain embedded parameter and metadata schemas as value trees
- Warn on catalog entries outside the provider's kind/codePrefix and
  on structural names matching no parameters property
- Publish action kinds, attachment levels, Result types as constants
- Bind an explicit concurrency null as absent; ConcurrencyCap carries
  a plain value
- Split files into a shared/flow/provider grid; move the flow graph
  onto a flowChecker
- Pin per-rule diagnostics in provider bind/check tables; run the
  provider corpus through ParseProvider

cmd/mwl:
- Wire mwl parse provider (--ast, --strict) via a shared runner
- Encode Provider through the reflection walker; drop encConcurrency
- Reorganize testdata into flow/ and provider/; add provider goldens

docs:
- Record decisions 15 (published constants) and 16 (empty-spelling
  normalization); sharpen decision 8's round-trip wording
- Rephrase narrated-history wording in both DESIGN documents
- Rewrite decision 11's test paragraphs: agreement between parser and
  schemas is held transitively through the shared fixture corpus by
  two suites (check-schemas.sh schema-side, the corpus tests
  parser-side), with no JSON Schema engine and a zero-dependency
  go.mod even for tests
- Record the accepted limit: agreement is only as wide as the corpus
- Add a per-segment negative lookahead to the call and middleware URI
  patterns in both schemas: the prose forbids "." and ".." as URI
  segments, but the charset alone admitted them (the parser already
  rejected them)
- Legal dot-bearing segments (v1.2, .hidden, "...") still match
- Add dot-segment invalid fixtures on both the flow and provider
  sides, encoding the divergence in the corpus
- ast_flow.go: replace block field comments with go/ast-style trailing
  comments, one clause per field; promote invariants and non-obvious
  semantics to the type doc comments
- ast_provider.go: same conversion for the provider AST
- ast.go: comment Expr.Body and the arms of the three-way fields
- diagnostic.go: convert Diagnostic.Code and .Path to trailing comments
- README.md: the decode/bind/check pipeline, the parse contract, a
  file map, and the deliberate boundaries; DESIGN.md keeps the why
- doc.go: name the three passes in the package comment; say "checks"
  rather than "findings" for analyses beyond the parser's scope
…er pages

- content/tools/: section landing page, parser overview, mwl CLI page,
  and the AST explorer — an interactive field-by-field reference for
  the Flow and Provider trees the parser returns
- scripts/check-explorer.py: drift check diffing the explorer page's
  data-field/data-type attributes against the parser AST declarations
- lefthook.yml: run the explorer check pre-commit on staged go/md/py
- scripts/check-build.sh: widen the HTML anchor check to cover the
  tools section alongside the reference
- hugo.yaml, content/_index.md: add the Tooling menu entry and landing
  card between Reference and Rationale
- layouts/_default/list.specmarkdown.md: skip sections flagged
  singleFileExclude in the whole-site spec.md; content/tools sets it
- dprint.json: exclude ast-explorer.md (raw interactive HTML rows)
- CLAUDE.md: document the section, the drift check, the SpecMarkdown
  exclusion, and the Python 3.10 floor for check-explorer.py
- Publish the action discriminator as Step.Action(), one method per
  variant; drop the CLI's type-to-action table and its panic path
- Extract bindResultType for the non-success type check both
  constructed failures share
- Share the invalid-provider-URI message via a binder helper
- Unify the per-pass diagnostic accumulators into one reporter in
  diagnostic.go; delete check.go and the decoder's bespoke copy
- Key the AST encoder's policy tables by (type, field) instead of bare
  field name; keep Comment as an AST-wide convention beside Pos
- Amend both DESIGN.md files and the parser README accordingly
- Standardize generic tool references on "the MWL CLI"; reserve
  backticked mwl for literal command and binary contexts
- Record the casing rule in CLAUDE.md's Casing section
- Set cmd/mwl/README.md's heading in code font
- Align the home page's Tooling card subtitle
@pjhartzell
pjhartzell requested a review from jkeifer as a code owner July 25, 2026 02:47
@pjhartzell pjhartzell changed the title Add parser and CLI Add a parser Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant