refactor(argtype): move the parser and printer upstream to @argtype/core - #85
Open
nx10 wants to merge 2 commits into
Open
refactor(argtype): move the parser and printer upstream to @argtype/core#85nx10 wants to merge 2 commits into
nx10 wants to merge 2 commits into
Conversation
Styx neither parses nor prints argtype any more. Both ends go through
@argtype/core, the language's reference implementation, released
independently of this compiler.
- Frontend: parseArgtype -> inlineAliases -> resolveAnnotations upstream,
then lower.ts turns the resolved document into IR.
- Backend: emit.ts builds an AstDocument and hands it to printArgtype.
Deletes lexer.ts, parser.ts, ast.ts, doc.ts, frontmatter.ts and
template.ts (~1.9k lines). This repo now holds no knowledge of argtype
syntax - no quoting, no escaping, no template metacharacters, no layout -
only the IR correspondence, which is the part it is qualified to decide.
The split follows what each side is allowed to decide: a parser must keep
the document intact, a generator may narrow it. Lowering is where the
narrowing happens (`set` -> sequence, `any` -> first branch, aliases
inlined), so it stays here rather than upstream, where it would be wrong
for a validator, a runner or an editor.
Extensions are opt-in by import rather than by config flag.
resolveAnnotations interprets the spec core only; resolveOutputs,
resolveMediaTypes and resolvePaths are separate upstream passes keyed by
node. resolveConstraints is deliberately not run - the IR cannot express
inter-argument rules. An annotation no imported module claims is reported
by lower.ts rather than dropped, so a .requires() never vanishes from a
generated wrapper without a signal.
Two behaviour changes fall out of the split:
- Upstream target errors now fire everywhere, including on nodes lowering
discards. `any("-f", x: str.mutable())` is an error even though only
branch 0 reaches the IR; suppressing it would make the same file valid
or invalid depending on the order of its `any` branches.
- An output scope is a sequence regardless of how many children the
wrapper has. A lone child used to be returned bare, so its outputs
escaped the wrapper that gates them and the generated wrapper promised
them unconditionally; adding one unrelated literal moved them back
inside and flipped the generated type.
There is no typo detection any more: after the extension split no layer
knows the whole universe of method names, so a misspelled .min() and a
method from an unimplemented extension are indistinguishable. Both
surface as the same lowering warning.
Verified: 1300 tests pass, tsc/eslint/prettier clean, and the niwrap
corpus round-trips 1879/1879.
…ed on
Review follow-up to the @argtype/core migration. One regression, one masked
bug, and three places where a boundary was looser than it read.
`wrapChildren` wraps a lone child in a sequence when it carries outputs, so
the flag convention in the `opt` case (`inner.kind === "literal"`) stopped
matching and `.output()` on a bare flag silently turned `m?: boolean` into an
empty `param_2?: ToolParam2` - the parameter ceased to exist and a caller
enabled the flag by passing `{}`. Fixed in three parts, because the first two
alone each traded one break for another:
- `isFlagContent` accepts the scoping sequence, not just a bare literal.
- The solver treats `optional(struct{})` as a bool: an empty struct carries no
information beyond its own presence. Narrowed to `optional` - a union *arm*
that solves to an empty struct is discriminated by `@type`, which is the
`antsIntroduction.sh` `n4_correction` shape.
- `wrapChildren` lifts a lone child's name and doc onto the wrapper, which is
what the enclosing `opt`/`rep` is named after.
The end-to-end test that should have caught this compared the one-child
spelling against its two-child twin, and a multi-child flag is synthetically
named whether or not outputs are involved - so the baseline was already
degraded and the two never agreed. It now compares each spelling against the
same source with the `.output()` removed, which is the actual property: the
scoping sequence must not reach the caller.
`buildOptional`/`buildRepeat` tested `!inner.meta` to decide whether a
sequence spreads into its wrapper, so an *empty* meta object blocked the
spread: `opt(seq("-a"))` emitted nested, re-parsed flat, and the second emit
disagreed. That non-idempotence was excluded from the property generator as an
artifact of `simplify`, which that test never runs. The guard is gone, the
generator covers single-child sequences again, and the shape is pinned
deterministically - it surfaced in roughly one sample in ten thousand.
Boundary hardening, all of it invisible until an upstream release moves:
- Pin `@argtype/core` exactly. CI installs with `npm ci`, so a caret left the
version consumers actually resolve untested here - styx's lockfile is not
published and the hub installs against its own.
- `IMPLEMENTED_METHODS` is a Styx-owned list rather than a spread of the
upstream vocabularies. It answers "does Styx read this?", and deriving it
from "does upstream declare this?" meant a method added to `CORE_METHODS`
would join the set with no reader behind it and silently stop being
reported. `implemented-methods.test.ts` fails when upstream grows a name.
- Diagnostic severity switches explicitly and degrades an unknown level to a
warning. `error` as the fallback would turn a purely additive upstream
level into a hard compile failure, and errors fail the whole build
downstream.
Also corrects the cross-repo canary claim. `corpus-roundtrip` asserts only
zero-errors on compile and re-parse; it never compares direct-compile IR
against re-parsed IR, so it is a syntax smoke test and near-blind to a
semantic regression.
Verified: 1307 tests pass, tsc/eslint/prettier clean, codegen typecheck
(tsc + mypy --strict) passes, corpus round-trips 1879/1879.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Styx neither parses nor prints argtype any more. Both ends go through
@argtype/core, the language's reference implementation, released independently of this compiler.parseArgtype→inlineAliases→resolveAnnotationsupstream, thenfrontend/argtype/lower.tsturns the resolved document into IR.backend/argtype/emit.tsbuilds anAstDocumentand hands it toprintArgtype.This deletes
lexer.ts,parser.ts,ast.ts,doc.ts,frontmatter.tsandtemplate.ts- about 1.9k lines. The repo now holds no knowledge of argtype syntax: no quoting, no escaping, no template metacharacters, no layout. Only the IR correspondence, which is the part it is actually qualified to decide.Why the line falls here
A parser must keep the document intact; a generator may narrow it. Lowering is where the narrowing happens -
setbecomes a sequence,anybecomes its first branch (lossless when emitting one invocation, lossy for anything that parses argv), aliases are inlined by substitution. Those choices are correct here and wrong for a validator, a runner, or an editor, so they live in this repo rather than upstream.Extensions are opt-in by import, not by a config flag.
resolveAnnotationsinterprets the spec core only; each extension vocabulary is a separate upstream pass (resolveOutputs,resolveMediaTypes,resolvePaths) returning results keyed by node, whichparser-frontend.tsruns and passes tolowerDocumentasLoweringExtensions.resolveConstraintsis deliberately not run - the IR cannot express inter-argument rules. An annotation no imported module claims is reported bylower.ts(IMPLEMENTED_METHODS) rather than dropped; a.requires()must never vanish from a generated wrapper without a signal.Behaviour changes worth reviewing
A diagnostic describes the document, not the subtree lowering keeps. Two halves of one rule that pull in opposite directions if taken separately:
any("-f", x: str.mutable())is an error even though only branch 0 reaches the IR -.mutable()on astris invalid argtype wherever it is written. It was only ever accepted because the old check ran per-lowered-node and never saw the branch. Suppressing it would make the same file valid or invalid depending on the order of itsanybranches.lower.ts's unclaimed-method scan walks the whole resolved document (reportUnconsumed), and distinguishes "Styx cannot represent this method" from "this node is not in the wrapper, so the method is ignored here".An output scope is a
sequence, and how many children a wrapper happens to have is not part of that.opt/repwith several children implicitly wrap them in a sequence that owns their.output()s; a lone child used to be returned bare with the enclosing sink passed through, so its outputs escaped the wrapper that gates them and the generated wrapper promised them unconditionally. Adding one unrelated literal moved them back inside and flipped the generated type - which is what marked it a bug rather than a convention. It has to be asequenceand not justmeta.outputson the child: only a sequence is force-bound as an output scope (solver.ts), andsimplifykeeps a single-literal sequence alive precisely when it carries outputs.No typo detection any more. After the extension split no layer knows the whole universe of method names, so a misspelled
.min()and a method from an extension Styx does not implement are the same thing; both surface as the same lowering warning.CORE_METHODS/OUTPUT_METHODS/ … are exported upstream if that ever needs reviving here.One asymmetry:
printArgtypeemits a///block line for line, because a printer reproduces the source it was given.emit.tstherefore word-wraps a description itself before handing it over - when generating from IR there is no original layout to preserve, so choosing one is the emitter's job.Release coupling
@argtype/coremust be published before merging a PR that raises the pin inpackages/core/package.json, ornpm cifails on an unpublished version.0.1.0is on npm, so this one is fine. Thecorpus-roundtripjob doubles as the cross-repo canary - a regression on either side surfaces as descriptors that stop round-tripping.Local trap:
scripts/corpus-roundtrip.mjsimportspackages/core/dist, notsrc, so a staledisttests the previous code and passes while the working tree is broken. Alwaysnpm run build -w @styx-api/core -w @styx-api/clifirst.Verification
npx vitest run- 1300 passed, 5 skippednpx tsc --noEmit- cleaneslint+prettier --check- cleannpm run corpus:roundtrip -- --catalog ../niwrap/src/niwrap- 1879/1879Review follow-up (
d26dbfd)A four-lens review found one regression introduced by this PR, one bug it masked, and three boundary issues. All fixed in the follow-up commit.
The output-scope fix broke the flag it was declared on.
wrapChildrenwraps a lone child in a sequence when it carries outputs, so the flag convention in theoptcase (inner.kind === "literal") stopped matching:.output()on a bare flag turnedm?: booleaninto an emptyparam_2?: ToolParam2, and a caller enabled the flag by passing{}. It needed three parts, because the first two each traded one break for another:isFlagContentaccepts the scoping sequence, not just a bare literal.optional(struct{})as a bool. Narrowed tooptional: a union arm that solves to an empty struct is discriminated by@type, which is theantsIntroduction.shn4_correctionshape.wrapChildrenlifts a lone child's name and doc onto the wrapper.The end-to-end test that should have caught this compared the one-child spelling against its two-child twin. A multi-child flag is synthetically named whether or not outputs are involved, so that baseline was already degraded and the two spellings never agreed. It now compares each spelling against the same source with the
.output()removed, which is the real property: the scoping sequence must not reach the caller.A masked non-idempotence.
buildOptional/buildRepeattested!inner.metato decide whether a sequence spreads into its wrapper, so an empty meta object blocked the spread:opt(seq("-a"))emitted nested, re-parsed flat, second emit disagreed. That was excluded from the property generator as an artifact ofsimplify, which that test never runs. Guard removed, generator covers single-child sequences again, shape pinned deterministically (it surfaced in roughly one sample in ten thousand).Boundary hardening, all invisible until an upstream release moves:
@argtype/corepinned exactly. CI installs withnpm ci, so a caret left the version consumers actually resolve untested here: styx's lockfile is not published and the hub installs against its own.IMPLEMENTED_METHODSis Styx-owned rather than a spread of the upstream vocabularies. It answers "does Styx read this?", and deriving it from "does upstream declare this?" meant a method added toCORE_METHODSwould join the set with no reader behind it and silently stop being reported.implemented-methods.test.tsfails when upstream grows a name.errorand turning an additive upstream level into a hard build failure downstream.Corrected claim:
corpus-roundtripis not the cross-repo canary this PR originally advertised. It asserts only zero-errors on compile and re-parse, never comparing direct-compile IR against re-parsed IR, so it is a syntax smoke test and near-blind to a semantic regression.ARCHITECTURE.mdnow says what each gate actually proves.Verification: 1307 tests pass, tsc/eslint/prettier clean, codegen typecheck (tsc +
mypy --strict) passes, corpus 1879/1879.Known and deliberately not fixed here
.output()written on theopt/repnode itself (rather than on a child) still escapes the wrapper that gates it, so__fixtures__/bet.argtype:38typesbinary_masknon-nullable. Not a regression (mainbehaves the same), but it means the two spellings remain inconsistent in the opposite direction. Worth its own change.