Skip to content

refactor(argtype): move the parser and printer upstream to @argtype/core - #85

Open
nx10 wants to merge 2 commits into
mainfrom
refactor/argtype-parser-package
Open

refactor(argtype): move the parser and printer upstream to @argtype/core#85
nx10 wants to merge 2 commits into
mainfrom
refactor/argtype-parser-package

Conversation

@nx10

@nx10 nx10 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Styx neither parses nor prints argtype any more. Both ends go through @argtype/core, the language's reference implementation, released independently of this compiler.

  • Frontend: parseArgtypeinlineAliasesresolveAnnotations upstream, then frontend/argtype/lower.ts turns the resolved document into IR.
  • Backend: backend/argtype/emit.ts builds an AstDocument and hands it to printArgtype.

This deletes lexer.ts, parser.ts, ast.ts, doc.ts, frontmatter.ts and template.ts - about 1.9k lines. The repo now holds no knowledge of argtype syntax: no quoting, no escaping, no template metacharacters, no layout. Only the IR correspondence, which is the part it is actually qualified to decide.

Why the line falls here

A parser must keep the document intact; a generator may narrow it. Lowering is where the narrowing happens - set becomes a sequence, any becomes its first branch (lossless when emitting one invocation, lossy for anything that parses argv), aliases are inlined by substitution. Those choices are correct here and wrong for a validator, a runner, or an editor, so they live in this repo rather than upstream.

Extensions are opt-in by import, not by a config flag. resolveAnnotations interprets the spec core only; each extension vocabulary is a separate upstream pass (resolveOutputs, resolveMediaTypes, resolvePaths) returning results keyed by node, which parser-frontend.ts runs and passes to lowerDocument as LoweringExtensions. resolveConstraints is deliberately not run - the IR cannot express inter-argument rules. An annotation no imported module claims is reported by lower.ts (IMPLEMENTED_METHODS) rather than dropped; a .requires() must never vanish from a generated wrapper without a signal.

Behaviour changes worth reviewing

A diagnostic describes the document, not the subtree lowering keeps. Two halves of one rule that pull in opposite directions if taken separately:

  • Upstream target errors now fire everywhere, including on nodes lowering discards. any("-f", x: str.mutable()) is an error even though only branch 0 reaches the IR - .mutable() on a str is invalid argtype wherever it is written. It was only ever accepted because the old check ran per-lowered-node and never saw the branch. Suppressing it would make the same file valid or invalid depending on the order of its any branches.
  • lower.ts's unclaimed-method scan walks the whole resolved document (reportUnconsumed), and distinguishes "Styx cannot represent this method" from "this node is not in the wrapper, so the method is ignored here".

An output scope is a sequence, and how many children a wrapper happens to have is not part of that. opt/rep with several children implicitly wrap them in a sequence that owns their .output()s; a lone child used to be returned bare with the enclosing sink passed through, so its outputs escaped the wrapper that gates them and the generated wrapper promised them unconditionally. Adding one unrelated literal moved them back inside and flipped the generated type - which is what marked it a bug rather than a convention. It has to be a sequence and not just meta.outputs on the child: only a sequence is force-bound as an output scope (solver.ts), and simplify keeps a single-literal sequence alive precisely when it carries outputs.

No typo detection any more. After the extension split no layer knows the whole universe of method names, so a misspelled .min() and a method from an extension Styx does not implement are the same thing; both surface as the same lowering warning. CORE_METHODS / OUTPUT_METHODS / … are exported upstream if that ever needs reviving here.

One asymmetry: printArgtype emits a /// block line for line, because a printer reproduces the source it was given. emit.ts therefore word-wraps a description itself before handing it over - when generating from IR there is no original layout to preserve, so choosing one is the emitter's job.

Release coupling

@argtype/core must be published before merging a PR that raises the pin in packages/core/package.json, or npm ci fails on an unpublished version. 0.1.0 is on npm, so this one is fine. The corpus-roundtrip job doubles as the cross-repo canary - a regression on either side surfaces as descriptors that stop round-tripping.

Local trap: scripts/corpus-roundtrip.mjs imports packages/core/dist, not src, so a stale dist tests the previous code and passes while the working tree is broken. Always npm run build -w @styx-api/core -w @styx-api/cli first.

Verification

  • npx vitest run - 1300 passed, 5 skipped
  • npx tsc --noEmit - clean
  • eslint + prettier --check - clean
  • npm run corpus:roundtrip -- --catalog ../niwrap/src/niwrap - 1879/1879

Review follow-up (d26dbfd)

A four-lens review found one regression introduced by this PR, one bug it masked, and three boundary issues. All fixed in the follow-up commit.

The output-scope fix broke the flag it was declared on. wrapChildren wraps a lone child in a sequence when it carries outputs, so the flag convention in the opt case (inner.kind === "literal") stopped matching: .output() on a bare flag turned m?: boolean into an empty param_2?: ToolParam2, and a caller enabled the flag by passing {}. It needed three parts, because the first two each traded one break for another:

  • isFlagContent accepts the scoping sequence, not just a bare literal.
  • The solver treats optional(struct{}) as a bool. Narrowed to optional: a union arm that solves to an empty struct is discriminated by @type, which is the antsIntroduction.sh n4_correction shape.
  • wrapChildren lifts a lone child's name and doc onto the wrapper.

The end-to-end test that should have caught this compared the one-child spelling against its two-child twin. A multi-child flag is synthetically named whether or not outputs are involved, so that baseline was already degraded and the two spellings never agreed. It now compares each spelling against the same source with the .output() removed, which is the real property: the scoping sequence must not reach the caller.

A masked non-idempotence. buildOptional/buildRepeat tested !inner.meta to decide whether a sequence spreads into its wrapper, so an empty meta object blocked the spread: opt(seq("-a")) emitted nested, re-parsed flat, second emit disagreed. That was excluded from the property generator as an artifact of simplify, which that test never runs. Guard removed, generator covers single-child sequences again, shape pinned deterministically (it surfaced in roughly one sample in ten thousand).

Boundary hardening, all invisible until an upstream release moves:

  • @argtype/core pinned exactly. CI installs with npm ci, so a caret left the version consumers actually resolve untested here: styx's lockfile is not published and the hub installs against its own.
  • IMPLEMENTED_METHODS is Styx-owned rather than a spread of the upstream vocabularies. It answers "does Styx read this?", and deriving it from "does upstream declare this?" meant a method added to CORE_METHODS would join the set with no reader behind it and silently stop being reported. implemented-methods.test.ts fails when upstream grows a name.
  • Diagnostic severity switches explicitly and degrades an unknown level to a warning, rather than falling back to error and turning an additive upstream level into a hard build failure downstream.

Corrected claim: corpus-roundtrip is not the cross-repo canary this PR originally advertised. It asserts only zero-errors on compile and re-parse, never comparing direct-compile IR against re-parsed IR, so it is a syntax smoke test and near-blind to a semantic regression. ARCHITECTURE.md now says what each gate actually proves.

Verification: 1307 tests pass, tsc/eslint/prettier clean, codegen typecheck (tsc + mypy --strict) passes, corpus 1879/1879.

Known and deliberately not fixed here

.output() written on the opt/rep node itself (rather than on a child) still escapes the wrapper that gates it, so __fixtures__/bet.argtype:38 types binary_mask non-nullable. Not a regression (main behaves the same), but it means the two spellings remain inconsistent in the opposite direction. Worth its own change.

nx10 added 2 commits August 5, 2026 12:09
Styx neither parses nor prints argtype any more. Both ends go through
@argtype/core, the language's reference implementation, released
independently of this compiler.

- Frontend: parseArgtype -> inlineAliases -> resolveAnnotations upstream,
  then lower.ts turns the resolved document into IR.
- Backend: emit.ts builds an AstDocument and hands it to printArgtype.

Deletes lexer.ts, parser.ts, ast.ts, doc.ts, frontmatter.ts and
template.ts (~1.9k lines). This repo now holds no knowledge of argtype
syntax - no quoting, no escaping, no template metacharacters, no layout -
only the IR correspondence, which is the part it is qualified to decide.

The split follows what each side is allowed to decide: a parser must keep
the document intact, a generator may narrow it. Lowering is where the
narrowing happens (`set` -> sequence, `any` -> first branch, aliases
inlined), so it stays here rather than upstream, where it would be wrong
for a validator, a runner or an editor.

Extensions are opt-in by import rather than by config flag.
resolveAnnotations interprets the spec core only; resolveOutputs,
resolveMediaTypes and resolvePaths are separate upstream passes keyed by
node. resolveConstraints is deliberately not run - the IR cannot express
inter-argument rules. An annotation no imported module claims is reported
by lower.ts rather than dropped, so a .requires() never vanishes from a
generated wrapper without a signal.

Two behaviour changes fall out of the split:

- Upstream target errors now fire everywhere, including on nodes lowering
  discards. `any("-f", x: str.mutable())` is an error even though only
  branch 0 reaches the IR; suppressing it would make the same file valid
  or invalid depending on the order of its `any` branches.
- An output scope is a sequence regardless of how many children the
  wrapper has. A lone child used to be returned bare, so its outputs
  escaped the wrapper that gates them and the generated wrapper promised
  them unconditionally; adding one unrelated literal moved them back
  inside and flipped the generated type.

There is no typo detection any more: after the extension split no layer
knows the whole universe of method names, so a misspelled .min() and a
method from an unimplemented extension are indistinguishable. Both
surface as the same lowering warning.

Verified: 1300 tests pass, tsc/eslint/prettier clean, and the niwrap
corpus round-trips 1879/1879.
…ed on

Review follow-up to the @argtype/core migration. One regression, one masked
bug, and three places where a boundary was looser than it read.

`wrapChildren` wraps a lone child in a sequence when it carries outputs, so
the flag convention in the `opt` case (`inner.kind === "literal"`) stopped
matching and `.output()` on a bare flag silently turned `m?: boolean` into an
empty `param_2?: ToolParam2` - the parameter ceased to exist and a caller
enabled the flag by passing `{}`. Fixed in three parts, because the first two
alone each traded one break for another:

- `isFlagContent` accepts the scoping sequence, not just a bare literal.
- The solver treats `optional(struct{})` as a bool: an empty struct carries no
  information beyond its own presence. Narrowed to `optional` - a union *arm*
  that solves to an empty struct is discriminated by `@type`, which is the
  `antsIntroduction.sh` `n4_correction` shape.
- `wrapChildren` lifts a lone child's name and doc onto the wrapper, which is
  what the enclosing `opt`/`rep` is named after.

The end-to-end test that should have caught this compared the one-child
spelling against its two-child twin, and a multi-child flag is synthetically
named whether or not outputs are involved - so the baseline was already
degraded and the two never agreed. It now compares each spelling against the
same source with the `.output()` removed, which is the actual property: the
scoping sequence must not reach the caller.

`buildOptional`/`buildRepeat` tested `!inner.meta` to decide whether a
sequence spreads into its wrapper, so an *empty* meta object blocked the
spread: `opt(seq("-a"))` emitted nested, re-parsed flat, and the second emit
disagreed. That non-idempotence was excluded from the property generator as an
artifact of `simplify`, which that test never runs. The guard is gone, the
generator covers single-child sequences again, and the shape is pinned
deterministically - it surfaced in roughly one sample in ten thousand.

Boundary hardening, all of it invisible until an upstream release moves:

- Pin `@argtype/core` exactly. CI installs with `npm ci`, so a caret left the
  version consumers actually resolve untested here - styx's lockfile is not
  published and the hub installs against its own.
- `IMPLEMENTED_METHODS` is a Styx-owned list rather than a spread of the
  upstream vocabularies. It answers "does Styx read this?", and deriving it
  from "does upstream declare this?" meant a method added to `CORE_METHODS`
  would join the set with no reader behind it and silently stop being
  reported. `implemented-methods.test.ts` fails when upstream grows a name.
- Diagnostic severity switches explicitly and degrades an unknown level to a
  warning. `error` as the fallback would turn a purely additive upstream
  level into a hard compile failure, and errors fail the whole build
  downstream.

Also corrects the cross-repo canary claim. `corpus-roundtrip` asserts only
zero-errors on compile and re-parse; it never compares direct-compile IR
against re-parsed IR, so it is a syntax smoke test and near-blind to a
semantic regression.

Verified: 1307 tests pass, tsc/eslint/prettier clean, codegen typecheck
(tsc + mypy --strict) passes, corpus round-trips 1879/1879.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant