From 271d05e7bbb07dd4b88f004b669c72e9da909051 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sun, 9 Aug 2026 05:20:08 +0000 Subject: [PATCH 1/6] docs(conformance): ADR 0002, extend the fixture contract to a case tree MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eight of the twenty-four diagnostic codes the three spec.md files declare are about a document's surroundings rather than its contents — a directory with a name, a sibling document, a file on disk. A case was one `case.yaml`, so none of them had a fixture, and they reached `main` as prose with CI green. That is the failure mode CONTRIBUTING.md ground rule 2 exists to prevent. GOVERNANCE.md lists "Changing the conformance fixture contract" among the changes needing an ADR. This is that change. A case directory MAY now carry a `tree/` instead of a `case.yaml`, with `metadata.document` naming the file under test inside it. `tree/` is the parent of the item root rather than the item root itself: ERR_SLUG_MISMATCH tests the item directory's *name*, so a fixture for it needs a directory that has one. `case.yaml` is not thereby the legacy form, and the ADR says so. It asserts that the document has no item root — a real state, per blueprint §3.1, and the one an API payload arrives in. Making `tree/` the only form would have traded one blind spot for another. Symlinks are declared in metadata rather than committed. ERR_PATH_ESCAPE is reachable only by symlink now that listing §5's grammar makes `..` unspellable, and a committed link does not survive a checkout without `core.symlinks`, is invisible in a diff, and would ship inside a release tarball pointing outside the archive. The ADR also corrects tools/src/validator.ts, which asserted that the semantic phase was out of scope "per ADR 0001". ADR 0001 §6 forbids *publishing* a reference validator and is untouched; §7 describes tools/ as one non-normative adapter. Implementing more phases inside it publishes nothing. The alternative was a corpus in which every semantic clause was checked by nothing at all. Refs: #9 Signed-off-by: Justin Merrell --- docs/adr/0002-conformance-case-trees.md | 170 ++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 docs/adr/0002-conformance-case-trees.md diff --git a/docs/adr/0002-conformance-case-trees.md b/docs/adr/0002-conformance-case-trees.md new file mode 100644 index 0000000..6a865cf --- /dev/null +++ b/docs/adr/0002-conformance-case-trees.md @@ -0,0 +1,170 @@ +# ADR 0002: Conformance case trees + +- **Status:** Accepted +- **Date:** 2026-08-09 +- **Extends:** [ADR 0001](0001-canonical-repository-architecture.md) §7 + +## Context + +A conformance case is one `case.yaml`. That was enough while the corpus covered +the `parser` and `structural` phases, both of which decide a document by +reading the document. + +The `semantic` phase does not. Of the twenty-four diagnostic codes the three +`spec.md` files now declare, eight are about a document's *surroundings*: + +| Code | Needs | +|---|---| +| `ERR_SLUG_MISMATCH` | a directory with a name | +| `ERR_VERSION_MISMATCH` | a sibling document | +| `ERR_UNREFERENCED_COMPONENT` | the item's other files | +| `ERR_COMPONENT_NOT_FOUND`, `ERR_REFERENCE_ESCAPE` | a resolvable target | +| `ERR_UNKNOWN_OUTPUT`, `ERR_CONFLICTING_INPUT_SCHEMA` | the referenced component documents | +| `ERR_MEDIA_NOT_FOUND`, `ERR_PATH_ESCAPE` | a file on disk | + +None is expressible as a single document, so none had a fixture. They reached +`main` as prose with CI green, which is the failure mode +[CONTRIBUTING.md](../../CONTRIBUTING.md) ground rule 2 exists to prevent: "no +schema change without conformance fixtures." + +The rules themselves are not in doubt. Blueprint §3.1 defines the **item root** +and every one of these codes is measured against it. What was missing is a way +for a fixture to *have* an item root. + +GOVERNANCE.md lists "Changing the conformance fixture contract" among the +changes needing an ADR. This is that change. + +## Decision + +### 1. A case directory MAY carry a `tree/` instead of a `case.yaml` + +The two forms are mutually exclusive, and the distinction is normative rather +than a convenience. + +``` +conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/ + metadata.json + diagnostics.json + tree/ + acme-wiki/ + blueprint.yaml + listing.yaml + components/ + postgres.yaml +``` + +`metadata.document` names the document under test as a path relative to +`tree/` — here `acme-wiki/blueprint.yaml`. The **item root** is the directory +containing that document. + +`tree/` is the parent of the item root, not the item root itself. The extra +level is load-bearing: `ERR_SLUG_MISMATCH` compares `metadata.slug` against the +item directory's *name*, so a fixture that tested it needs a directory it can +name. A tree that was itself the item root could only ever be called `tree`. + +### 2. `case.yaml` keeps its meaning, and it is not "the old form" + +Blueprint §3.1 says a document handed over without a directory has no item root, +and that an implementation in that position MUST NOT report any rule measured +against one. That is a real state — a document submitted over an API — and +`case.yaml` is how the corpus expresses it. + +So a `case.yaml` fixture asserts more than "these are the contents". It asserts +*there is no item root*, and an adapter that invents one for it is wrong. A case +declaring both forms is malformed. + +**Rejected:** making `tree/` the only form and giving every case a synthetic +item root. It would have made the no-directory rule untestable, replacing one +blind spot with another. + +### 3. Symlinks are declared in metadata, not committed + +`ERR_PATH_ESCAPE` is, after listing §5's grammar landed, reachable only by +symlink: `..` is unspellable, so a media path escapes the item root only when a +legal spelling resolves to an illegal target. + +A committed symlink is a poor fixture. It does not survive a Windows checkout +without `core.symlinks`, it is invisible in a diff, and a corpus shipped in a +release tarball would carry a link pointing outside the archive. + +`metadata.symlinks` declares them instead: + +```json +{ + "symlinks": { "acme-wiki/media/icon.png": "../../../../etc/passwd" } +} +``` + +Keys are paths relative to `tree/`; values are the link targets, verbatim. An +adapter materialises the tree into a scratch location and creates the links +there. The target is deliberately not required to exist — a dangling link that +resolves outside the item root is still an escape, and containment is decided +on the resolved path. + +**Rejected:** committing the symlinks with a `.gitattributes` exemption. It +moves the portability problem rather than solving it, and the fixture stops +being readable as data. + +**Rejected:** a `capability`-style skip for path escape. The rule is decided +offline against the filesystem, which is what makes it `semantic`; declining to +test it would have been a statement about this repository's tooling, not about +the contract. + +### 4. An adapter that cannot materialise a tree SKIPs, and MUST NOT pass + +This is the rule the README already states for an unimplemented phase, extended +to an unsupported case shape. A tree case is skipped, reported as skipped, and +never reported as passed. + +### 5. `tools/` implements the `semantic` phase + +`tools/src/validator.ts` previously asserted that the `semantic` phase "belongs +to a shared implementation library … not in scope for this repository, per +ADR 0001". That reading is wrong and this ADR corrects it. + +ADR 0001 §6 forbids publishing a reference validator: no binary, no shared +library, no WebAssembly module, no language bindings. ADR 0001 §7 separately +describes `tools/` as one non-normative adapter over the corpus, confined to its +own directory. Implementing more phases inside that adapter publishes nothing +and blesses nothing. It only means the fixtures in this repository are executed +rather than declared. + +The alternative was a corpus in which every `semantic` clause — acyclicity, the +floating-tag blocklist, the media rules — was checked by nothing at all. A +specification whose own CI cannot tell whether its fixtures are right is in a +worse position than one that admits a language affinity in a directory already +marked non-normative. + +`capability` stays unimplemented. It needs an account, a region and a quota, +which is a server, and no amount of local tooling substitutes for one. + +## Consequences + +**Positive** + +- The eight tree-shaped codes become testable, and the corpus can state what it + covers without an asterisk. +- The `semantic` phase is executed in CI, so a clause and its fixture can + disagree loudly instead of quietly. +- The no-item-root rule gains a fixture shape of its own, having previously been + the accidental default. + +**Negative** + +- Every downstream adapter must now handle two case shapes, and materialise a + tree with symlinks to run the full corpus. An adapter may skip the tree cases + and still be honest about it, but it is not conformant while it does. +- Release tarballs grow. `release.yml` already ships + `conformance//`; it now ships directories inside it. +- `tools/` carries more non-normative code, and more of it is the kind that + could drift from the prose. The reverse coverage gate added alongside this ADR + — every declared diagnostic code needs a fixture or an allowlist entry stating + why not — is what keeps that drift visible. + +## Follow-ups + +1. Author the fixtures this ADR unblocks; `ERR_UNKNOWN_COMPONENT` stays + uncovered because it is `capability`. +2. Reconcile the three points where the specification and the platform + deliberately disagree — cycle detection, parameter-merge conflicts, and + whether a `SERVICE` must expose an endpoint — before v1 is declared stable. From 4ffbaef4cea542dacc6afff67b97e7248be25c30 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sun, 9 Aug 2026 05:20:33 +0000 Subject: [PATCH 2/6] fix(tools): reject YAML anchors and aliases in the parser phase MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Component §7 has said since bootstrap that the parser phase MUST reject anchors and aliases. Nothing did. `parseDocument` set `uniqueKeys`, `merge: false` and `strict`, none of which touches an alias — a document spelling `&b` / `*b` parsed cleanly and the alias expanded. The parser now walks the tree and rejects both. `maxAliasCount` would have covered the alias and not the lone anchor, which is inert and rejected anyway: an author who writes one is reaching for a feature this contract withholds, and §2 already settled that a field which does nothing is an error rather than a silence. Two codes rather than one. `ERR_ANCHOR_OR_ALIAS` is new. `ERR_INVALID_YAML` is not — `validator.ts` has emitted it since bootstrap and no spec.md declared it, so a fixture citing it would have been rejected by the runner's own registry check as an unknown code. Both are now in component §8, and the two say different things to an author: one means the document is malformed, the other means it is well-formed and uses something this contract does not have. Also switches the parser from `parse` to `parseDocument`, which reports every problem instead of throwing on the first and carries machine-readable error codes. The previous implementation matched on message text — the one thing this specification explicitly declares non-normative. The parser lives in a new document.ts so that semantic.ts can read an item's other documents without closing an import cycle. Fail-before evidence: all six anchor/alias cases validate cleanly against tools/ at be77e19 and fail after. The three malformed-YAML cases are green beforehand and are the regression pin for the code registration. This tightens the parser phase — a document with an alias validates today and will not after. It is free while no tag exists and release PRs #1/#2/#3 are open; after they merge it is a v2 directory plus a migration note. Refs: #9 Signed-off-by: Justin Merrell --- .../002-reject-anchors-and-aliases/case.yaml | 19 +++++ .../diagnostics.json | 6 ++ .../metadata.json | 7 ++ .../parser/003-reject-lone-anchor/case.yaml | 14 ++++ .../003-reject-lone-anchor/diagnostics.json | 6 ++ .../003-reject-lone-anchor/metadata.json | 7 ++ .../v1/parser/004-malformed-yaml/case.yaml | 7 ++ .../004-malformed-yaml/diagnostics.json | 6 ++ .../parser/004-malformed-yaml/metadata.json | 7 ++ .../002-reject-anchors-and-aliases/case.yaml | 20 +++++ .../diagnostics.json | 6 ++ .../metadata.json | 7 ++ .../parser/003-reject-lone-anchor/case.yaml | 17 ++++ .../003-reject-lone-anchor/diagnostics.json | 6 ++ .../003-reject-lone-anchor/metadata.json | 7 ++ .../v1/parser/004-malformed-yaml/case.yaml | 7 ++ .../004-malformed-yaml/diagnostics.json | 6 ++ .../parser/004-malformed-yaml/metadata.json | 7 ++ .../002-reject-anchors-and-aliases/case.yaml | 15 ++++ .../diagnostics.json | 6 ++ .../metadata.json | 7 ++ .../parser/003-reject-lone-anchor/case.yaml | 13 ++++ .../003-reject-lone-anchor/diagnostics.json | 6 ++ .../003-reject-lone-anchor/metadata.json | 7 ++ .../v1/parser/004-malformed-yaml/case.yaml | 7 ++ .../004-malformed-yaml/diagnostics.json | 6 ++ .../parser/004-malformed-yaml/metadata.json | 7 ++ specifications/component/v1/spec.md | 24 +++++- tools/src/document.ts | 73 ++++++++++++++++++ tools/src/validator.ts | Bin 4719 -> 5075 bytes 30 files changed, 332 insertions(+), 1 deletion(-) create mode 100644 conformance/blueprint/v1/parser/002-reject-anchors-and-aliases/case.yaml create mode 100644 conformance/blueprint/v1/parser/002-reject-anchors-and-aliases/diagnostics.json create mode 100644 conformance/blueprint/v1/parser/002-reject-anchors-and-aliases/metadata.json create mode 100644 conformance/blueprint/v1/parser/003-reject-lone-anchor/case.yaml create mode 100644 conformance/blueprint/v1/parser/003-reject-lone-anchor/diagnostics.json create mode 100644 conformance/blueprint/v1/parser/003-reject-lone-anchor/metadata.json create mode 100644 conformance/blueprint/v1/parser/004-malformed-yaml/case.yaml create mode 100644 conformance/blueprint/v1/parser/004-malformed-yaml/diagnostics.json create mode 100644 conformance/blueprint/v1/parser/004-malformed-yaml/metadata.json create mode 100644 conformance/component/v1/parser/002-reject-anchors-and-aliases/case.yaml create mode 100644 conformance/component/v1/parser/002-reject-anchors-and-aliases/diagnostics.json create mode 100644 conformance/component/v1/parser/002-reject-anchors-and-aliases/metadata.json create mode 100644 conformance/component/v1/parser/003-reject-lone-anchor/case.yaml create mode 100644 conformance/component/v1/parser/003-reject-lone-anchor/diagnostics.json create mode 100644 conformance/component/v1/parser/003-reject-lone-anchor/metadata.json create mode 100644 conformance/component/v1/parser/004-malformed-yaml/case.yaml create mode 100644 conformance/component/v1/parser/004-malformed-yaml/diagnostics.json create mode 100644 conformance/component/v1/parser/004-malformed-yaml/metadata.json create mode 100644 conformance/listing/v1/parser/002-reject-anchors-and-aliases/case.yaml create mode 100644 conformance/listing/v1/parser/002-reject-anchors-and-aliases/diagnostics.json create mode 100644 conformance/listing/v1/parser/002-reject-anchors-and-aliases/metadata.json create mode 100644 conformance/listing/v1/parser/003-reject-lone-anchor/case.yaml create mode 100644 conformance/listing/v1/parser/003-reject-lone-anchor/diagnostics.json create mode 100644 conformance/listing/v1/parser/003-reject-lone-anchor/metadata.json create mode 100644 conformance/listing/v1/parser/004-malformed-yaml/case.yaml create mode 100644 conformance/listing/v1/parser/004-malformed-yaml/diagnostics.json create mode 100644 conformance/listing/v1/parser/004-malformed-yaml/metadata.json create mode 100644 tools/src/document.ts diff --git a/conformance/blueprint/v1/parser/002-reject-anchors-and-aliases/case.yaml b/conformance/blueprint/v1/parser/002-reject-anchors-and-aliases/case.yaml new file mode 100644 index 0000000..878d237 --- /dev/null +++ b/conformance/blueprint/v1/parser/002-reject-anchors-and-aliases/case.yaml @@ -0,0 +1,19 @@ +# Expanded, this document is entirely valid: two nodes, neither wired to +# anything. It is rejected anyway — the rule is about the spelling, not about +# what the spelling produces. +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 1 +spec: + components: + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: &unwired {} + web: + component: ./components/web.yaml + size: general.standard.small + connections: *unwired + parameters: {} diff --git a/conformance/blueprint/v1/parser/002-reject-anchors-and-aliases/diagnostics.json b/conformance/blueprint/v1/parser/002-reject-anchors-and-aliases/diagnostics.json new file mode 100644 index 0000000..18394ba --- /dev/null +++ b/conformance/blueprint/v1/parser/002-reject-anchors-and-aliases/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_ANCHOR_OR_ALIAS", + "path": "" + } +] diff --git a/conformance/blueprint/v1/parser/002-reject-anchors-and-aliases/metadata.json b/conformance/blueprint/v1/parser/002-reject-anchors-and-aliases/metadata.json new file mode 100644 index 0000000..86d60f5 --- /dev/null +++ b/conformance/blueprint/v1/parser/002-reject-anchors-and-aliases/metadata.json @@ -0,0 +1,7 @@ +{ + "id": "parser-002-reject-anchors-and-aliases", + "phase": "parser", + "expected": "fail", + "clause": "specifications/blueprint/v1/spec.md#envelope", + "summary": "A document using a YAML anchor and alias is rejected before structural validation." +} diff --git a/conformance/blueprint/v1/parser/003-reject-lone-anchor/case.yaml b/conformance/blueprint/v1/parser/003-reject-lone-anchor/case.yaml new file mode 100644 index 0000000..53beb5f --- /dev/null +++ b/conformance/blueprint/v1/parser/003-reject-lone-anchor/case.yaml @@ -0,0 +1,14 @@ +# An anchor nothing refers to, on a document that is valid in every other +# respect. A rule written against aliases alone lets it through. +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 1 +spec: + components: + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: &unwired {} + parameters: {} diff --git a/conformance/blueprint/v1/parser/003-reject-lone-anchor/diagnostics.json b/conformance/blueprint/v1/parser/003-reject-lone-anchor/diagnostics.json new file mode 100644 index 0000000..18394ba --- /dev/null +++ b/conformance/blueprint/v1/parser/003-reject-lone-anchor/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_ANCHOR_OR_ALIAS", + "path": "" + } +] diff --git a/conformance/blueprint/v1/parser/003-reject-lone-anchor/metadata.json b/conformance/blueprint/v1/parser/003-reject-lone-anchor/metadata.json new file mode 100644 index 0000000..9eb5c25 --- /dev/null +++ b/conformance/blueprint/v1/parser/003-reject-lone-anchor/metadata.json @@ -0,0 +1,7 @@ +{ + "id": "parser-003-reject-lone-anchor", + "phase": "parser", + "expected": "fail", + "clause": "specifications/blueprint/v1/spec.md#envelope", + "summary": "An anchor with no alias is rejected, not merely an alias." +} diff --git a/conformance/blueprint/v1/parser/004-malformed-yaml/case.yaml b/conformance/blueprint/v1/parser/004-malformed-yaml/case.yaml new file mode 100644 index 0000000..674c1e1 --- /dev/null +++ b/conformance/blueprint/v1/parser/004-malformed-yaml/case.yaml @@ -0,0 +1,7 @@ +# A tab where the indentation should be. YAML 1.2 forbids tabs as +# indentation, and the document never becomes a mapping — so no later phase +# has anything to inspect, which is why the pipeline stops here. +specVersion: v1 +kind: BLUEPRINT +metadata: + version: 1 diff --git a/conformance/blueprint/v1/parser/004-malformed-yaml/diagnostics.json b/conformance/blueprint/v1/parser/004-malformed-yaml/diagnostics.json new file mode 100644 index 0000000..a287506 --- /dev/null +++ b/conformance/blueprint/v1/parser/004-malformed-yaml/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_INVALID_YAML", + "path": "" + } +] diff --git a/conformance/blueprint/v1/parser/004-malformed-yaml/metadata.json b/conformance/blueprint/v1/parser/004-malformed-yaml/metadata.json new file mode 100644 index 0000000..3adaa41 --- /dev/null +++ b/conformance/blueprint/v1/parser/004-malformed-yaml/metadata.json @@ -0,0 +1,7 @@ +{ + "id": "parser-004-malformed-yaml", + "phase": "parser", + "expected": "fail", + "clause": "specifications/blueprint/v1/spec.md#envelope", + "summary": "A document that is not well-formed YAML 1.2 is rejected in the parser phase." +} diff --git a/conformance/component/v1/parser/002-reject-anchors-and-aliases/case.yaml b/conformance/component/v1/parser/002-reject-anchors-and-aliases/case.yaml new file mode 100644 index 0000000..6abdcf0 --- /dev/null +++ b/conformance/component/v1/parser/002-reject-anchors-and-aliases/case.yaml @@ -0,0 +1,20 @@ +# Expanded, this document is entirely valid: two probes, both polling +# `/healthz`. It is rejected anyway, and that is the point — the rule is about +# the spelling, not about what the spelling produces. +# +# A reader who has not expanded the alias cannot see that changing `readiness` +# also changes `liveness`. +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: nginx:1.29.4-alpine + health: + readiness: &probe + path: /healthz + liveness: *probe diff --git a/conformance/component/v1/parser/002-reject-anchors-and-aliases/diagnostics.json b/conformance/component/v1/parser/002-reject-anchors-and-aliases/diagnostics.json new file mode 100644 index 0000000..18394ba --- /dev/null +++ b/conformance/component/v1/parser/002-reject-anchors-and-aliases/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_ANCHOR_OR_ALIAS", + "path": "" + } +] diff --git a/conformance/component/v1/parser/002-reject-anchors-and-aliases/metadata.json b/conformance/component/v1/parser/002-reject-anchors-and-aliases/metadata.json new file mode 100644 index 0000000..b44794c --- /dev/null +++ b/conformance/component/v1/parser/002-reject-anchors-and-aliases/metadata.json @@ -0,0 +1,7 @@ +{ + "id": "parser-002-reject-anchors-and-aliases", + "phase": "parser", + "expected": "fail", + "clause": "specifications/component/v1/spec.md#envelope", + "summary": "A document using a YAML anchor and alias is rejected before structural validation." +} diff --git a/conformance/component/v1/parser/003-reject-lone-anchor/case.yaml b/conformance/component/v1/parser/003-reject-lone-anchor/case.yaml new file mode 100644 index 0000000..3434aee --- /dev/null +++ b/conformance/component/v1/parser/003-reject-lone-anchor/case.yaml @@ -0,0 +1,17 @@ +# An anchor nothing refers to. It expands to nothing, changes no value, and +# leaves a document that is valid in every other respect — so a rule written +# against aliases alone lets it through. +# +# It is rejected because an author who writes one is reaching for a feature this +# contract withholds, and finding that out now beats finding it out when the +# alias is added. +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: &image + type: IMAGE + ref: nginx:1.29.4-alpine diff --git a/conformance/component/v1/parser/003-reject-lone-anchor/diagnostics.json b/conformance/component/v1/parser/003-reject-lone-anchor/diagnostics.json new file mode 100644 index 0000000..18394ba --- /dev/null +++ b/conformance/component/v1/parser/003-reject-lone-anchor/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_ANCHOR_OR_ALIAS", + "path": "" + } +] diff --git a/conformance/component/v1/parser/003-reject-lone-anchor/metadata.json b/conformance/component/v1/parser/003-reject-lone-anchor/metadata.json new file mode 100644 index 0000000..95ecc7d --- /dev/null +++ b/conformance/component/v1/parser/003-reject-lone-anchor/metadata.json @@ -0,0 +1,7 @@ +{ + "id": "parser-003-reject-lone-anchor", + "phase": "parser", + "expected": "fail", + "clause": "specifications/component/v1/spec.md#envelope", + "summary": "An anchor with no alias is rejected, not merely an alias." +} diff --git a/conformance/component/v1/parser/004-malformed-yaml/case.yaml b/conformance/component/v1/parser/004-malformed-yaml/case.yaml new file mode 100644 index 0000000..3dc0b56 --- /dev/null +++ b/conformance/component/v1/parser/004-malformed-yaml/case.yaml @@ -0,0 +1,7 @@ +# A tab where the indentation should be. YAML 1.2 forbids tabs as +# indentation, and the document never becomes a mapping — so no later phase +# has anything to inspect, which is why the pipeline stops here. +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 diff --git a/conformance/component/v1/parser/004-malformed-yaml/diagnostics.json b/conformance/component/v1/parser/004-malformed-yaml/diagnostics.json new file mode 100644 index 0000000..a287506 --- /dev/null +++ b/conformance/component/v1/parser/004-malformed-yaml/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_INVALID_YAML", + "path": "" + } +] diff --git a/conformance/component/v1/parser/004-malformed-yaml/metadata.json b/conformance/component/v1/parser/004-malformed-yaml/metadata.json new file mode 100644 index 0000000..53d889c --- /dev/null +++ b/conformance/component/v1/parser/004-malformed-yaml/metadata.json @@ -0,0 +1,7 @@ +{ + "id": "parser-004-malformed-yaml", + "phase": "parser", + "expected": "fail", + "clause": "specifications/component/v1/spec.md#envelope", + "summary": "A document that is not well-formed YAML 1.2 is rejected in the parser phase." +} diff --git a/conformance/listing/v1/parser/002-reject-anchors-and-aliases/case.yaml b/conformance/listing/v1/parser/002-reject-anchors-and-aliases/case.yaml new file mode 100644 index 0000000..560c129 --- /dev/null +++ b/conformance/listing/v1/parser/002-reject-anchors-and-aliases/case.yaml @@ -0,0 +1,15 @@ +# Expanded, this document is entirely valid: the description repeats the +# tagline. It is rejected anyway — the rule is about the spelling, not about +# what the spelling produces. +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: COMPONENT + displayName: Acme Wiki + summary: &tagline A wiki backed by PostgreSQL + description: *tagline + category: PRODUCTIVITY + lifecycleStage: STABLE diff --git a/conformance/listing/v1/parser/002-reject-anchors-and-aliases/diagnostics.json b/conformance/listing/v1/parser/002-reject-anchors-and-aliases/diagnostics.json new file mode 100644 index 0000000..18394ba --- /dev/null +++ b/conformance/listing/v1/parser/002-reject-anchors-and-aliases/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_ANCHOR_OR_ALIAS", + "path": "" + } +] diff --git a/conformance/listing/v1/parser/002-reject-anchors-and-aliases/metadata.json b/conformance/listing/v1/parser/002-reject-anchors-and-aliases/metadata.json new file mode 100644 index 0000000..282af87 --- /dev/null +++ b/conformance/listing/v1/parser/002-reject-anchors-and-aliases/metadata.json @@ -0,0 +1,7 @@ +{ + "id": "parser-002-reject-anchors-and-aliases", + "phase": "parser", + "expected": "fail", + "clause": "specifications/listing/v1/spec.md#envelope", + "summary": "A document using a YAML anchor and alias is rejected before structural validation." +} diff --git a/conformance/listing/v1/parser/003-reject-lone-anchor/case.yaml b/conformance/listing/v1/parser/003-reject-lone-anchor/case.yaml new file mode 100644 index 0000000..ee8b19b --- /dev/null +++ b/conformance/listing/v1/parser/003-reject-lone-anchor/case.yaml @@ -0,0 +1,13 @@ +# An anchor nothing refers to, on a document that is valid in every other +# respect. A rule written against aliases alone lets it through. +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: COMPONENT + displayName: Acme Wiki + summary: &tagline A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE diff --git a/conformance/listing/v1/parser/003-reject-lone-anchor/diagnostics.json b/conformance/listing/v1/parser/003-reject-lone-anchor/diagnostics.json new file mode 100644 index 0000000..18394ba --- /dev/null +++ b/conformance/listing/v1/parser/003-reject-lone-anchor/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_ANCHOR_OR_ALIAS", + "path": "" + } +] diff --git a/conformance/listing/v1/parser/003-reject-lone-anchor/metadata.json b/conformance/listing/v1/parser/003-reject-lone-anchor/metadata.json new file mode 100644 index 0000000..3340d14 --- /dev/null +++ b/conformance/listing/v1/parser/003-reject-lone-anchor/metadata.json @@ -0,0 +1,7 @@ +{ + "id": "parser-003-reject-lone-anchor", + "phase": "parser", + "expected": "fail", + "clause": "specifications/listing/v1/spec.md#envelope", + "summary": "An anchor with no alias is rejected, not merely an alias." +} diff --git a/conformance/listing/v1/parser/004-malformed-yaml/case.yaml b/conformance/listing/v1/parser/004-malformed-yaml/case.yaml new file mode 100644 index 0000000..41fb74d --- /dev/null +++ b/conformance/listing/v1/parser/004-malformed-yaml/case.yaml @@ -0,0 +1,7 @@ +# A tab where the indentation should be. YAML 1.2 forbids tabs as +# indentation, and the document never becomes a mapping — so no later phase +# has anything to inspect, which is why the pipeline stops here. +specVersion: v1 +kind: LISTING +metadata: + version: 1 diff --git a/conformance/listing/v1/parser/004-malformed-yaml/diagnostics.json b/conformance/listing/v1/parser/004-malformed-yaml/diagnostics.json new file mode 100644 index 0000000..a287506 --- /dev/null +++ b/conformance/listing/v1/parser/004-malformed-yaml/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_INVALID_YAML", + "path": "" + } +] diff --git a/conformance/listing/v1/parser/004-malformed-yaml/metadata.json b/conformance/listing/v1/parser/004-malformed-yaml/metadata.json new file mode 100644 index 0000000..f75f667 --- /dev/null +++ b/conformance/listing/v1/parser/004-malformed-yaml/metadata.json @@ -0,0 +1,7 @@ +{ + "id": "parser-004-malformed-yaml", + "phase": "parser", + "expected": "fail", + "clause": "specifications/listing/v1/spec.md#envelope", + "summary": "A document that is not well-formed YAML 1.2 is rejected in the parser phase." +} diff --git a/specifications/component/v1/spec.md b/specifications/component/v1/spec.md index 5fb02b5..1d63b3f 100644 --- a/specifications/component/v1/spec.md +++ b/specifications/component/v1/spec.md @@ -294,6 +294,26 @@ pass. A client MUST NOT require network access for the `parser`, `structural`, or `semantic` phases. +**Why the `parser` phase rejects legal YAML.** A duplicate key, an anchor and +an alias are all well-formed YAML 1.2, and all three are rejected here. + +A duplicate key has no defined winner: parsers disagree on whether the first or +the last survives, so a document carrying one means two things. An alias means +one thing, but only after expansion, and a document whose meaning depends on +being expanded is not readable as the thing it declares — the same reason +[§2](#envelope) rejects a misspelled optional field rather than ignoring it. An +anchor with no alias is inert, and is rejected anyway: an author who writes one +is reaching for a feature this contract does not have, and finding that out at +authoring time is better than finding it out when the alias is added. + +The bound also matters. Alias expansion is where a small document becomes a +large one — the billion-laughs shape — and a validator that must expand before +it can measure has no way to refuse cheaply. + +`ERR_ANCHOR_OR_ALIAS` is separate from `ERR_INVALID_YAML` because the two say +different things to an author. One means the document is malformed; the other +means it is well-formed and uses something this contract withholds. + ## 8. Diagnostics Diagnostic **codes** and the **phase** at which validation fails are normative. @@ -302,7 +322,9 @@ different text and that is expected. | Code | Phase | Meaning | |---|---|---| +| `ERR_INVALID_YAML` | `parser` | The document is not well-formed YAML 1.2. | | `ERR_DUPLICATE_KEY` | `parser` | The same mapping key appears twice. | +| `ERR_ANCHOR_OR_ALIAS` | `parser` | The document declares a YAML anchor or an alias. | | `ERR_UNSUPPORTED_SPEC_VERSION` | `structural` | `specVersion` is not a supported value. | | `ERR_WRONG_KIND` | `structural` | `kind` does not match the family being validated. | | `ERR_UNKNOWN_FIELD` | `structural` | A property not defined by the schema is present. | @@ -312,7 +334,7 @@ different text and that is expected. | `ERR_UNPINNED_IMAGE` | `semantic` | An image reference carries a floating tag. | | `ERR_UNKNOWN_ENDPOINT` | `semantic` | A probe names an endpoint the workload does not declare. | -The rows above the `semantic` pair are the shared envelope registry: the +The `parser` and `structural` rows are the shared envelope registry: the [blueprint](../../blueprint/v1/spec.md#diagnostics) and [listing](../../listing/v1/spec.md#diagnostics) families declare themselves additions to this table rather than restating it. The two `semantic` codes are diff --git a/tools/src/document.ts b/tools/src/document.ts new file mode 100644 index 0000000..3b709e9 --- /dev/null +++ b/tools/src/document.ts @@ -0,0 +1,73 @@ +/** + * The document types every phase shares, and the `parser` phase itself. + * + * Separate from validator.ts so that semantic.ts can read the component and + * listing documents an item contains without importing the module that imports + * it. NON-NORMATIVE, like everything under tools/. + */ +import { isNode, parseDocument as parseYamlDocument, visit } from 'yaml' +import type { Json } from './spec.ts' + +export type Phase = 'parser' | 'structural' | 'semantic' | 'capability' + +export interface Diagnostic { + /** Normative. Implementations map their internal errors onto these. */ + readonly code: string + /** Normative. JSON Pointer into the document, `` for the root. */ + readonly path: string + /** Non-normative — text differs between implementations by design. */ + readonly message: string +} + +/** + * Strict YAML 1.2. Duplicate keys, merge keys, anchors and aliases are all + * rejected outright — rules JSON Schema cannot express, so they must be + * enforced before it runs. Component §7 is the clause. + * + * `parseDocument` rather than `parse` because it collects every problem instead + * of throwing on the first, and because its errors carry machine-readable codes. + * The previous implementation matched on message text, which is exactly the + * thing this specification declares non-normative. + */ +export function parseDocument(source: string): { value: Json } | { errors: Diagnostic[] } { + const doc = parseYamlDocument(source, { + uniqueKeys: true, + merge: false, + strict: true, + version: '1.2', + }) + + const errors: Diagnostic[] = doc.errors.map((error) => ({ + code: error.code === 'DUPLICATE_KEY' ? 'ERR_DUPLICATE_KEY' : 'ERR_INVALID_YAML', + path: '', + message: error.message, + })) + + // An alias expands and an anchor is inert, and both are rejected. The reason + // is §2's: an author who writes one believes it does something, and a document + // whose meaning depends on which parser expands it is not one contract. + // + // Walked rather than configured because no parser option covers a lone anchor + // — `maxAliasCount` only ever sees the alias. + visit(doc, { + Alias(_key, node) { + errors.push({ + code: 'ERR_ANCHOR_OR_ALIAS', + path: '', + message: `alias *${node.source} is not permitted`, + }) + }, + Node(_key, node) { + if (isNode(node) && node.anchor !== undefined) { + errors.push({ + code: 'ERR_ANCHOR_OR_ALIAS', + path: '', + message: `anchor &${node.anchor} is not permitted`, + }) + } + }, + }) + + if (errors.length > 0) return { errors } + return { value: doc.toJS() as Json } +} diff --git a/tools/src/validator.ts b/tools/src/validator.ts index 123cbf3516bac8798be1c24e926df605ff93eecd..573db66e1bb3b3234ba18b8b0c8f24c96407e54f 100644 GIT binary patch delta 1896 zcmY*a%Wfk@6jd|`kP(`!&}={#3CR=WX=fP0NJd#80)hn(B|&UmuD(u}Qm*Q$s*Y_% zQNteq<^$Nx2h1M~AAo>A;0M^V;@s-CldNo2cb$9gIp^N~_r1S|Kbdw!)0je|QxlY( zl5!;uc%fV}dP<>jRB_;frdri%W~@o;DV3fhIiDs2dY?=p$6Q9-C8f1*TpE$0z~{!} z1b~X3H9n~kEi0odGI*6Dhf5Ac3j_f$6hxJc8RWO9{hra#klN`iWG+fv#5PYbk*HF?G66ayTmKlwRQZaM!bde#`ZCw zHoxsX*!;eC&|cjkl)2FLU_7Q1e#9%O#wA5hT3O2>(gkxZI#N#so@CK#Q?oS=5j%qx zxtY)fK)@`7Mi3k#*EI+V&`GPdiM*pNhg}8{#^j;5Lh2l%$%`S}gE>LawjjMn9TQO+ z*RpJqbz1tymZYsm1x0$TEf<5XB?#Ed{aUa0#@_qu#h(W7j0pA#d$Yq7<$LW2oOo$IDKu^Ys zDYZ*a7seswfZR128i+^Vn^gHG(*T7Lc-nQoC6DtqfC=_S280HVvo<9hfFm;ek|)Us zD))Oqb!}nxV>G^n@*beFE3iyBOlW2st}#i{@{{5*MYWbbui3`udwQ!Ry$|u0pr8U==a#$mh8- zE+#Rppf7V*%3@$-NlbOc`eMQ>C57TNn+m1=`Qg{>%J#uH+aXI*p?~q{7EdF~q#qku zpxZ@xlZ3eexB$w=T5xF;1VkGpCUVANE_^C|-om^xAd9sqroi!qL}a|3i+Qv8@&1D^ z21IkUFm^qm&-3pHC($=S^F)RLHp@vLstKLmT|{(5SJ$J#!DQ3kdu#K@{l|Opm;KM) z%p+G89D)yVbGrZJ6*H%Yo1gdJZ4c-Qco`AwcCkC!{k^r(9(L6Fjae4>U^ED0Pv#)w y1jZ_M7;}FC!acjfvO7mSj*gD#5Z7hzPH2e7Z|CpiaD>ckmvg+i_u{|(X77LD;+wbt delta 1459 zcmZWp(QX?>6cw~Z)KyUeqJo#|R3L2}jFWSk;bLXD9=iGDWpXk32f01(tm0)O{DPmU` zkqu!Ag_I1n`IvH{oQ#JUQKr?z!f6QDR2WJc*oMrf8Rev$a8fH|a%6-l7nre_ijmBu zE7#IpO6VR8@bAyRLeXcE_Xg5JDi3I5Jtd5_flK+&&{SLLv?*Jd^1W{F5WBm(_p3Kv zkH4G>V+q0{ERIEgKhN<_D=@kS`Dg z5EJ17(WM%TZLLDsT(XK94SF6Pkjjx6i%uT9SCoUxvRHMJ~q zF`Y9{k}QCSP$B~vqm6}DtTcx#r)}>rlp&F5m6eEfLLG34QcYfc;^EP^`#8`6&VvRJ zBswm9ms%JOYjFuiYuBD%L7xhjzJjhkR!b70ZH}HhByv0^u5?818IihT1wJp4kd+g4 zl|W9`ib>7k8k49#zyA09hzZ3G$K7xCaKHUY8@*zh$yhkTDU~)@Dkn2pZVeNlrxZI% zpbJi(q%IU^ij1tcQjp>(PPu!Cg;Lsq`(^m`7HrU^L|$0O3_y1jdoE_I5%Gc`ou30+ ziH$q+;EY}^;g;0}UtDU`?6AGgL{{*L_Y^zuf=@yhJ1^T|77XEdIJ_~yZB+bp$Yl4e z99+Y<>RUje=JKb493x5&H~6aiUH{}!|2V3HOo)RggZ&@6dxPGI zH_jroVpBKaJd6zgAHmvD)MDv^1DRx?`p1p4H}B$*zdipqE&hSvx66N^dU5;x>i3)P zR?jvz?~cPq(ph~d>yD`Y+89*tZT8=K9FUTBDC0@>bo0|&i>B*S)}wl{dG8)mkz;J> c)1|(v+VyGZV)ffQA6)O?@smG)e7kt Date: Sun, 9 Aug 2026 05:20:53 +0000 Subject: [PATCH 3/6] feat(tools): implement the semantic phase and the case-tree runner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every semantic clause PR #13 wrote — acyclicity, the floating-tag blocklist, ERR_CONFLICTING_INPUT_SCHEMA, the media rules — was prose that nothing in CI ever executed. Five fixtures existed and all five were skipped. New semantic.ts, split the way the prose splits the rules rather than the way this runner would find convenient. An **in-document** rule is decided by reading the document: a floating image tag, a probe naming an endpoint that is not declared, a `fromRole` naming no node, a cycle, two screenshots sharing a basename. An **item-scoped** rule is measured against the item root, and is reported only when the caller supplies one. That is blueprint §3.1's rule, not a concession: "an implementation handed a document rather than a directory MUST NOT report any of them … a diagnostic it cannot substantiate is worse than a silence." It is also why `check:examples` stays green — `examples/` is not an item root, and its documents reference `./components/postgres.yaml` and `media/icon.png`, neither of which exists beside them. PR #13's body flagged that as latent breakage; this is the shape that makes it not one. Cycle reporting follows §4.2's canonical form exactly: strongly connected components, each reported as a closed walk from the lexicographically smallest node, choosing the smallest successor at each step. Verified stable across the three key orderings of one three-node cycle — the walk is a property of the graph, not of the traversal, which is the whole point of pinning it. conformance.ts gains the ADR 0002 case-tree form: `tree/` materialised to a scratch directory with its declared symlinks created there, validated with the item root set to the document's own directory. A case must declare exactly one of `case.yaml` or `tree/`; declaring both leaves it ambiguous which one the diagnostics describe. `IMPLEMENTED_PHASES` gains `semantic`. `capability` stays out — it needs an account, a region and a quota, which is a server. Refs: #9 Signed-off-by: Justin Merrell --- tools/src/conformance.ts | 221 ++++++++++++- tools/src/semantic.ts | 669 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 879 insertions(+), 11 deletions(-) create mode 100644 tools/src/semantic.ts diff --git a/tools/src/conformance.ts b/tools/src/conformance.ts index fd0c1f6..73dde3e 100644 --- a/tools/src/conformance.ts +++ b/tools/src/conformance.ts @@ -12,8 +12,19 @@ * second kind runs for every case, including the ones the first kind skips, * because a `semantic` fixture would otherwise be checked by nothing at all. */ -import { existsSync, readFileSync } from 'node:fs' -import { join } from 'node:path' +import { + cpSync, + existsSync, + mkdirSync, + mkdtempSync, + readdirSync, + readFileSync, + rmSync, + statSync, + symlinkSync, +} from 'node:fs' +import { tmpdir } from 'node:os' +import { dirname, isAbsolute, join, normalize } from 'node:path' import { discoverFamilies, Failures, @@ -38,6 +49,16 @@ interface CaseMetadata { readonly expected: 'pass' | 'fail' readonly clause?: string readonly summary?: string + /** + * Tree cases only (ADR 0002). Path of the document under test, relative to + * `tree/`. Its containing directory is the item root. + */ + readonly document?: string + /** + * Tree cases only. Link path (relative to `tree/`) to link target, verbatim. + * Materialised at run time rather than committed — see ADR 0002 §3. + */ + readonly symlinks?: Record } interface DeclaredDiagnostic { @@ -47,7 +68,13 @@ interface DeclaredDiagnostic { const PHASES: readonly Phase[] = ['parser', 'structural', 'semantic', 'capability'] -const IMPLEMENTED_PHASES = new Set(['parser', 'structural']) +/** + * `capability` is the one phase this repository cannot run: it needs an account, + * a region and a quota, which is a server. The other three are executed here — + * see ADR 0002 §5 for why running them is not this repository publishing a + * reference validator. + */ +const IMPLEMENTED_PHASES = new Set(['parser', 'structural', 'semantic']) /** * The family whose diagnostics table the other families declare themselves @@ -123,6 +150,93 @@ function loadIndex(family: Family, failures: Failures): CaseIndexEntry[] { return entries } +/** + * A path is safe to join onto a materialised tree when it is relative and stays + * inside it. Link *targets* are exempt — escaping is the thing some of them test + * — but the link's own location is not. + */ +function isContainedRelative(path: string): boolean { + return !isAbsolute(path) && !normalize(path).startsWith('..') +} + +/** + * ADR 0002 — a case declares its subject as exactly one of `case.yaml` (a + * document with no item root) or `tree/` (a document inside one). Declaring both + * would leave it ambiguous which one the diagnostics describe; declaring neither + * leaves nothing to validate. + */ +function checkCaseSubject( + caseDir: string, + label: string, + metadata: CaseMetadata, + failures: Failures, +): boolean { + const hasDocument = existsSync(join(caseDir, 'case.yaml')) + const hasTree = existsSync(join(caseDir, 'tree')) + + if (hasDocument === hasTree) { + failures.add( + `${label}: a case declares exactly one of case.yaml or tree/, not ${hasTree ? 'both' : 'neither'}`, + ) + return false + } + + if (!hasTree) { + if (metadata.document !== undefined || metadata.symlinks !== undefined) { + failures.add(`${label}: "document" and "symlinks" belong to a tree case`) + return false + } + return true + } + + let ok = true + if (metadata.document === undefined) { + failures.add(`${label}: a tree case must name its "document" relative to tree/`) + return false + } + if (!isContainedRelative(metadata.document)) { + failures.add(`${label}: "document" must be a relative path inside tree/`) + return false + } + if (!existsSync(join(caseDir, 'tree', metadata.document))) { + failures.add(`${label}: "document" names ${metadata.document}, which is not in tree/`) + ok = false + } + // tree/ is the *parent* of the item root, so that the item directory has a + // name a fixture can test ERR_SLUG_MISMATCH against (ADR 0002 §1). + if (dirname(metadata.document) === '.') { + failures.add(`${label}: "document" must sit inside an item directory under tree/`) + ok = false + } + for (const link of Object.keys(metadata.symlinks ?? {})) { + if (!isContainedRelative(link)) { + failures.add(`${label}: symlink "${link}" must be a relative path inside tree/`) + ok = false + } + } + return ok +} + +/** + * Copy a case's `tree/` somewhere writable and create its declared symlinks. + * Returns the scratch root; the caller removes it. + * + * Links are made here rather than committed because a committed one does not + * survive a checkout without `core.symlinks`, is invisible in a diff, and would + * ship inside a release tarball pointing outside the archive (ADR 0002 §3). + */ +function materialiseTree(caseDir: string, metadata: CaseMetadata): string { + const scratch = mkdtempSync(join(tmpdir(), 'musher-conformance-')) + cpSync(join(caseDir, 'tree'), scratch, { recursive: true }) + for (const [link, target] of Object.entries(metadata.symlinks ?? {})) { + const path = join(scratch, link) + mkdirSync(dirname(path), { recursive: true }) + rmSync(path, { force: true }) + symlinkSync(target, path) + } + return scratch +} + /** * Check the parts of a case that hold whether or not the phase runs here: the * declared outcome, the clause it traces to, and the codes it names. @@ -154,6 +268,7 @@ function checkCaseShape( failures.add(`${label}: id must follow -- and lead with the phase`) ok = false } + if (!checkCaseSubject(caseDir, label, metadata, failures)) ok = false // Every case should trace to prose — a fixture that cites nothing is an // assertion about an implementation, not about the specification. @@ -209,21 +324,43 @@ function checkCaseShape( return ok ? diagnostics : null } +/** + * Validate a case's subject, supplying an item root only when the case declares + * one. A `case.yaml` deliberately supplies none: blueprint §3.1 says a document + * arriving without a directory has no item root, and the rules measured against + * one MUST NOT be reported for it. + */ +function runValidation(family: Family, caseDir: string, metadata: CaseMetadata) { + if (metadata.document === undefined) { + return validateDocument(family, readFileSync(join(caseDir, 'case.yaml'), 'utf8')) + } + + const scratch = materialiseTree(caseDir, metadata) + try { + const documentPath = join(scratch, metadata.document) + return validateDocument(family, readFileSync(documentPath, 'utf8'), { + itemRoot: dirname(documentPath), + documentPath, + }) + } finally { + rmSync(scratch, { recursive: true, force: true }) + } +} + function runCase( family: Family, entry: CaseIndexEntry, failures: Failures, + /** Collects every code the corpus declares, for the coverage check. */ + exercised: Set, ): 'ran' | 'skipped' | 'failed' { const caseDir = join(family.conformanceDir, entry.path) const label = `${family.name}/${family.major}/${entry.id}` const metadataPath = join(caseDir, 'metadata.json') - const documentPath = join(caseDir, 'case.yaml') - for (const required of [metadataPath, documentPath]) { - if (!existsSync(required)) { - failures.add(`${label}: missing ${relativeToRepo(required)}`) - return 'failed' - } + if (!existsSync(metadataPath)) { + failures.add(`${label}: missing ${relativeToRepo(metadataPath)}`) + return 'failed' } const metadata = readJson(metadataPath) as unknown as CaseMetadata @@ -238,6 +375,7 @@ function runCase( const declared = checkCaseShape(family, entry, caseDir, label, metadata, failures) if (declared === null) return 'failed' + for (const item of declared) exercised.add(item.code) if (!IMPLEMENTED_PHASES.has(metadata.phase)) { console.log(` · ${label}: ${metadata.phase} phase not implemented here — skipped`) @@ -249,7 +387,7 @@ function runCase( return 'failed' } - const result = validateDocument(family, readFileSync(documentPath, 'utf8')) + const result = runValidation(family, caseDir, metadata) if (metadata.expected === 'pass') { if (result.ok) { @@ -286,6 +424,62 @@ function runCase( return 'ran' } +/** + * Diagnostic codes deliberately left without a fixture, and why. Every entry is + * a claim a reviewer can check; the list is short on purpose. + */ +const UNCOVERED: ReadonlyMap = new Map([ + [ + 'ERR_UNKNOWN_COMPONENT', + 'capability — resolving a published reference needs the catalog, and no phase this repository runs may reach the network', + ], +]) + +/** + * The reverse of `checkCaseShape`'s registry check. + * + * That one asks whether every code a fixture *declares* is defined by the prose. + * This one asks whether every code the prose *defines* is exercised by a + * fixture — the direction nothing enforced, and the direction ten codes reached + * `main` untested in with CI green. + * + * A code goes uncovered only by someone adding it to `UNCOVERED` with a reason, + * in a diff a reviewer sees. + */ +function checkCoverage(family: Family, exercised: ReadonlySet, failures: Failures): void { + // Only the family's own additions: a code inherited from component §8 is + // covered by component's corpus, and demanding a fixture per family would + // make every family restate the envelope suite. + const own = specIndex(family.specPath)?.codes ?? new Map() + for (const code of own.keys()) { + if (exercised.has(code) || UNCOVERED.has(code)) continue + failures.add( + `${family.name}/${family.major}: ${code} is declared in ${relativeToRepo(family.specPath)} ` + + 'but no indexed case exercises it. Add a fixture, or record it in UNCOVERED with a reason.', + ) + } +} + +/** + * Case directories that no `cases.json` entry names. The index is the contract + * and the runner never walks the filesystem, so an unindexed directory is not a + * failing fixture — it is an invisible one, which is worse. + */ +function checkOrphans(family: Family, entries: CaseIndexEntry[], failures: Failures): void { + const indexed = new Set(entries.map((entry) => normalize(entry.path))) + for (const phase of PHASES) { + const phaseDir = join(family.conformanceDir, phase) + if (!existsSync(phaseDir)) continue + for (const name of readdirSync(phaseDir)) { + if (!statSync(join(phaseDir, name)).isDirectory()) continue + if (indexed.has(normalize(join(phase, name)))) continue + failures.add( + `${family.name}/${family.major}: ${phase}/${name} is not indexed in cases.json and runs nowhere`, + ) + } + } +} + function main(): void { const failures = new Failures() let ran = 0 @@ -297,11 +491,16 @@ function main(): void { console.log(` · ${family.name}/${family.major}: no conformance cases indexed`) continue } + + const exercised = new Set() for (const entry of entries) { - const outcome = runCase(family, entry, failures) + const outcome = runCase(family, entry, failures, exercised) if (outcome === 'ran') ran += 1 if (outcome === 'skipped') skipped += 1 } + + checkCoverage(family, exercised, failures) + checkOrphans(family, entries, failures) } const suffix = skipped > 0 ? ` (${skipped} skipped)` : '' diff --git a/tools/src/semantic.ts b/tools/src/semantic.ts new file mode 100644 index 0000000..3522a4a --- /dev/null +++ b/tools/src/semantic.ts @@ -0,0 +1,669 @@ +/** + * The `semantic` phase: rules JSON Schema cannot express. + * + * NON-NORMATIVE, like everything under tools/. The definitive rule for each + * check below is the `spec.md` clause named in its comment; this file is one + * adapter's reading of it, and where the two disagree the prose wins. + * + * The rules divide by what they need, and the division is the prose's, not this + * runner's. An **in-document** rule is decided by reading the document. An + * **item-scoped** rule is measured against the item root + * ([blueprint §3.1](../../specifications/blueprint/v1/spec.md#item-directory)), + * and a caller that supplies no item root MUST NOT have those rules reported: + * "a diagnostic it cannot substantiate is worse than a silence." + */ +import { + existsSync, + lstatSync, + readdirSync, + readFileSync, + readlinkSync, + realpathSync, +} from 'node:fs' +import { basename, dirname, isAbsolute, join, relative, resolve } from 'node:path' +import { type Diagnostic, parseDocument } from './document.ts' +import { type Family, isObject, type Json } from './spec.ts' + +export type { Diagnostic } + +export interface SemanticContext { + /** + * Absolute path to the item root. Absent when the document did not arrive + * with a directory — a payload submitted over an API, or an `examples/` + * document. Item-scoped rules are silent without it. + */ + readonly itemRoot?: string + /** + * Absolute path to the document under test. Repo-local component references + * resolve relative to its directory (blueprint §4.1). Defaults to the item + * root's conventional `blueprint.yaml` when omitted. + */ + readonly documentPath?: string +} + +/** Escape one JSON Pointer reference token (RFC 6901 §3). */ +function token(value: string): string { + return value.replace(/~/g, '~0').replace(/\//g, '~1') +} + +function child(value: Json | undefined, key: string): Json | undefined { + return isObject(value) ? value[key] : undefined +} + +function asString(value: Json | undefined): string | undefined { + return typeof value === 'string' ? value : undefined +} + +/** Mapping keys in document order, or an empty list when the node is not one. */ +function keysOf(value: Json | undefined): string[] { + return isObject(value) ? Object.keys(value) : [] +} + +// =========================================================================== +// component +// =========================================================================== + +/** + * Component §5.1. Curated, and it will grow — which is exactly why it is here + * and not in a `pattern`. Growing a pattern makes a previously valid document + * invalid; growing this list is a minor release. + */ +const FLOATING_TAGS = new Set([ + 'latest', + 'main', + 'main-stable', + 'master', + 'stable', + 'edge', + 'nightly', + 'dev', + 'rolling', +]) + +/** + * The tag of an image reference, or `undefined` when it carries none. + * + * The tag colon is the one after the final slash. Without that rule + * `localhost:5000/nginx` reads as an image named `localhost` tagged + * `5000/nginx`, and a reference behind a ported registry is misjudged. + */ +function imageTag(ref: string): string | undefined { + const afterSlash = ref.slice(ref.lastIndexOf('/') + 1) + const colon = afterSlash.indexOf(':') + return colon === -1 ? undefined : afterSlash.slice(colon + 1) +} + +/** Component §5.1 — a reference MUST NOT carry a floating tag. */ +function checkImageRef(document: Json, out: Diagnostic[]): void { + const source = child(child(child(document, 'spec'), 'workload'), 'source') + const ref = asString(child(source, 'ref')) + if (ref === undefined || child(source, 'type') !== 'IMAGE') return + + // A digest is what resolves, so it satisfies the rule whatever tag it carries. + if (ref.includes('@sha256:')) return + + const tag = imageTag(ref) + if (tag !== undefined && FLOATING_TAGS.has(tag.toLowerCase())) { + out.push({ + code: 'ERR_UNPINNED_IMAGE', + path: '/spec/workload/source/ref', + message: `tag "${tag}" floats; pin a digest or an immutable tag`, + }) + } +} + +/** + * Component §5.4 — a probe's `endpoint` MUST name a declared endpoint. JSON + * Schema cannot: the names are mapping keys elsewhere in the document, and no + * keyword constrains a value against a sibling's keys. + */ +function checkProbeEndpoints(document: Json, out: Diagnostic[]): void { + const workload = child(child(document, 'spec'), 'workload') + const health = child(workload, 'health') + if (!isObject(health)) return + + const declared = new Set(keysOf(child(workload, 'endpoints'))) + for (const probe of keysOf(health)) { + // null selects the primary endpoint, so only a named one can be wrong. + const endpoint = asString(child(child(health, probe), 'endpoint')) + if (endpoint !== undefined && !declared.has(endpoint)) { + out.push({ + code: 'ERR_UNKNOWN_ENDPOINT', + path: `/spec/workload/health/${token(probe)}/endpoint`, + message: `probe targets endpoint "${endpoint}", which the workload does not declare`, + }) + } + } +} + +// =========================================================================== +// blueprint +// =========================================================================== + +const LOCAL_REFERENCE = /^\.\.?\// + +/** Consumer-anchored edges: node → the nodes it reads a value from. */ +function connectionGraph(document: Json): Map { + const components = child(child(document, 'spec'), 'components') + const graph = new Map() + for (const node of keysOf(components)) { + const connections = child(child(components, node), 'connections') + const targets: string[] = [] + for (const key of keysOf(connections)) { + const from = asString(child(child(connections, key), 'fromRole')) + if (from !== undefined) targets.push(from) + } + graph.set(node, targets) + } + return graph +} + +/** Blueprint §4.2 — `fromRole` MUST name a node in this blueprint. */ +function checkConnectionRoles(document: Json, out: Diagnostic[]): void { + const components = child(child(document, 'spec'), 'components') + const nodes = new Set(keysOf(components)) + for (const node of nodes) { + const connections = child(child(components, node), 'connections') + for (const key of keysOf(connections)) { + const from = asString(child(child(connections, key), 'fromRole')) + if (from !== undefined && !nodes.has(from)) { + out.push({ + code: 'ERR_UNKNOWN_ROLE', + path: `/spec/components/${token(node)}/connections/${token(key)}/fromRole`, + message: `fromRole "${from}" names no node in this blueprint`, + }) + } + } + } +} + +/** + * Blueprint §4.2 — the connection graph MUST be acyclic. + * + * Reporting is normative and pinned to a canonical form: the walk begins at the + * lexicographically smallest node in the cycle. Two implementations that find + * the same cycle then name it identically, instead of leaking whichever node + * their traversal happened to start from into a comparable diagnostic. + * + * Cycles are found as strongly connected components, then reported one per + * component. An SCC is the right granularity: a knot of four mutually reachable + * nodes is one problem, not one per elementary cycle through it. + */ +function checkCycles(document: Json, out: Diagnostic[]): void { + const graph = connectionGraph(document) + const nodes = [...graph.keys()].sort() + + // Iterative Tarjan — a blueprint is small, but a recursive walk would put the + // stack depth at the mercy of the document. + const index = new Map() + const low = new Map() + const onStack = new Set() + const stack: string[] = [] + const components: string[][] = [] + let counter = 0 + + for (const root of nodes) { + if (index.has(root)) continue + const work: { node: string; next: number }[] = [{ node: root, next: 0 }] + index.set(root, counter) + low.set(root, counter) + counter += 1 + stack.push(root) + onStack.add(root) + + while (work.length > 0) { + const frame = work[work.length - 1] + if (frame === undefined) break + const successors = (graph.get(frame.node) ?? []).filter((n) => graph.has(n)).sort() + + if (frame.next < successors.length) { + const successor = successors[frame.next] as string + frame.next += 1 + if (!index.has(successor)) { + index.set(successor, counter) + low.set(successor, counter) + counter += 1 + stack.push(successor) + onStack.add(successor) + work.push({ node: successor, next: 0 }) + } else if (onStack.has(successor)) { + low.set(frame.node, Math.min(low.get(frame.node) ?? 0, index.get(successor) ?? 0)) + } + continue + } + + work.pop() + const parent = work[work.length - 1] + if (parent !== undefined) { + low.set(parent.node, Math.min(low.get(parent.node) ?? 0, low.get(frame.node) ?? 0)) + } + if (low.get(frame.node) === index.get(frame.node)) { + const component: string[] = [] + for (;;) { + const popped = stack.pop() + if (popped === undefined) break + onStack.delete(popped) + component.push(popped) + if (popped === frame.node) break + } + components.push(component) + } + } + } + + for (const component of components) { + const members = new Set(component) + const selfLoop = + component.length === 1 && + (graph.get(component[0] as string) ?? []).includes(component[0] as string) + if (component.length < 2 && !selfLoop) continue + + const first = [...members].sort()[0] as string + out.push({ + code: 'ERR_DEPENDENCY_CYCLE', + path: `/spec/components/${token(first)}/connections`, + message: `connection cycle: ${closedWalk(graph, members, first).join(' -> ')}`, + }) + } +} + +/** + * A closed walk through `members` starting and ending at `first`, choosing the + * lexicographically smallest unvisited successor at each step so the walk is a + * property of the graph rather than of the traversal. + */ +function closedWalk(graph: Map, members: Set, first: string): string[] { + const walk = [first] + const seen = new Set([first]) + let current = first + for (;;) { + const successors = (graph.get(current) ?? []).filter((n) => members.has(n)).sort() + if (successors.includes(first) && walk.length > 1) break + const next = successors.find((n) => !seen.has(n)) + if (next === undefined) break + seen.add(next) + walk.push(next) + current = next + } + walk.push(first) + return walk +} + +// =========================================================================== +// listing +// =========================================================================== + +/** + * Listing §5 — two screenshots MUST NOT share a basename, across the whole item + * rather than within a directory. Published assets are addressed by basename, + * so `media/desktop/overview.png` and `media/mobile/overview.png` are one file. + */ +function checkScreenshotBasenames(document: Json, out: Diagnostic[]): void { + const screenshots = child(child(document, 'spec'), 'screenshots') + if (!Array.isArray(screenshots)) return + + const seen = new Map() + for (const [position, screenshot] of screenshots.entries()) { + const file = asString(child(screenshot, 'file')) + if (file === undefined) continue + const name = basename(file).toLowerCase() + const earlier = seen.get(name) + if (earlier === undefined) { + seen.set(name, position) + continue + } + // Reported at the later of the two: the first declaration is the one that + // stands, so the second is the one an author has to change. + out.push({ + code: 'ERR_DUPLICATE_MEDIA_BASENAME', + path: `/spec/screenshots/${position}/file`, + message: `basename "${basename(file)}" is already used by screenshot ${earlier}`, + }) + } +} + +// =========================================================================== +// item-scoped rules +// =========================================================================== + +/** Every media path a listing declares, paired with its JSON Pointer. */ +function mediaPaths(document: Json): { path: string; pointer: string }[] { + const spec = child(document, 'spec') + const found: { path: string; pointer: string }[] = [] + + const icon = asString(child(spec, 'icon')) + if (icon !== undefined) found.push({ path: icon, pointer: '/spec/icon' }) + + const screenshots = child(spec, 'screenshots') + if (Array.isArray(screenshots)) { + for (const [position, screenshot] of screenshots.entries()) { + const file = asString(child(screenshot, 'file')) + if (file !== undefined) { + found.push({ path: file, pointer: `/spec/screenshots/${position}/file` }) + } + } + } + return found +} + +/** + * True when `target` lies strictly inside `root`. Both are expected to be + * resolved already — this compares locations, and the caller is the one that + * decides what resolution means. + */ +function contains(root: string, target: string): boolean { + const rel = relative(root, target) + return rel !== '' && !rel.startsWith('..') && !isAbsolute(rel) +} + +/** + * The resolved location of `path`, following symlinks. Containment is a + * property of the resolved location rather than of the spelling — listing §5 + * and blueprint §4.1 both turn on that distinction. + */ +function resolveReal(path: string): string { + try { + return realpathSync(path) + } catch { + // A dangling symlink still has a target, and a target outside the item root + // is an escape whether or not anything is there. `readlinkSync` reads the + // link itself, which `realpathSync` cannot once the chain is broken. + try { + return resolve(dirname(path), readlinkSync(path)) + } catch { + // Not a link, or unreadable — the caller reports it as missing instead. + return resolve(path) + } + } +} + +/** Listing §5 — existence and containment, both of which need the filesystem. */ +function checkMediaOnDisk(document: Json, itemRoot: string, out: Diagnostic[]): void { + for (const { path, pointer } of mediaPaths(document)) { + const target = join(itemRoot, path) + if (!existsSync(target) && !isSymlink(target)) { + out.push({ + code: 'ERR_MEDIA_NOT_FOUND', + path: pointer, + message: `${path} does not exist in the item`, + }) + continue + } + if (!contains(itemRoot, resolveReal(target))) { + out.push({ + code: 'ERR_PATH_ESCAPE', + path: pointer, + message: `${path} resolves outside the item root`, + }) + } + } +} + +function isSymlink(path: string): boolean { + try { + return lstatSync(path).isSymbolicLink() + } catch { + return false + } +} + +/** Every `*.yaml`/`*.yml` under `root`, recursively, as absolute paths. */ +function yamlFiles(root: string): string[] { + const found: string[] = [] + const walk = (dir: string): void => { + for (const entry of readdirSync(dir, { withFileTypes: true })) { + const path = join(dir, entry.name) + if (entry.isDirectory()) walk(path) + else if (/\.ya?ml$/i.test(entry.name)) found.push(path) + } + } + walk(root) + return found.sort() +} + +const documentCache = new Map() + +/** Parse a document off disk, or `undefined` when it is unreadable. */ +function readDocument(path: string): Json | undefined { + if (documentCache.has(path)) return documentCache.get(path) + let parsed: Json | undefined + try { + const result = parseDocument(readFileSync(path, 'utf8')) + parsed = 'value' in result ? result.value : undefined + } catch { + // Unreadable or malformed. The item's own parser-phase run reports that; + // this one declines to describe a document it could not read. + parsed = undefined + } + documentCache.set(path, parsed) + return parsed +} + +/** + * Blueprint §3 and listing §3 — `metadata.slug` MUST equal the item directory + * name, and `metadata.version` MUST equal the sibling document's. + */ +function checkIdentity(family: Family, document: Json, itemRoot: string, out: Diagnostic[]): void { + const metadata = child(document, 'metadata') + const slug = asString(child(metadata, 'slug')) + const directory = basename(itemRoot) + if (slug !== undefined && slug !== directory) { + out.push({ + code: 'ERR_SLUG_MISMATCH', + path: '/metadata/slug', + message: `slug "${slug}" disagrees with the item directory "${directory}"`, + }) + } + + // The sibling is the other half of the item. Listing §3 conditions the rule + // on there being one: a `listingKind: COMPONENT` item need not hold a + // blueprint, and where there is none the rule has nothing to compare. + const siblingName = family.name === 'blueprint' ? 'listing.yaml' : 'blueprint.yaml' + const siblingPath = join(itemRoot, siblingName) + if (!existsSync(siblingPath)) return + + const sibling = readDocument(siblingPath) + const version = child(metadata, 'version') + const siblingVersion = child(child(sibling, 'metadata'), 'version') + if (sibling !== undefined && version !== siblingVersion) { + out.push({ + code: 'ERR_VERSION_MISMATCH', + path: '/metadata/version', + message: `version ${String(version)} disagrees with ${siblingName}'s ${String(siblingVersion)}`, + }) + } +} + +/** + * Blueprint §4.1 and §4.2, plus §3's unreferenced-document rule and §5.2's + * merge conflict. All four need the component documents the graph names, which + * is what makes them item-scoped. + */ +function checkGraphAgainstItem( + document: Json, + itemRoot: string, + documentPath: string, + out: Diagnostic[], +): void { + const components = child(child(document, 'spec'), 'components') + const baseDir = dirname(documentPath) + const referenced = new Set() + const resolved = new Map() + + for (const node of keysOf(components)) { + const reference = asString(child(child(components, node), 'component')) + const pointer = `/spec/components/${token(node)}/component` + // A published reference is a UUID and belongs to the capability phase; only + // the repo-local form resolves offline (§4.1). + if (reference === undefined || !LOCAL_REFERENCE.test(reference)) continue + + const target = resolve(baseDir, reference) + if (!contains(itemRoot, resolveReal(target))) { + out.push({ + code: 'ERR_REFERENCE_ESCAPE', + path: pointer, + message: `${reference} resolves outside the item root`, + }) + continue + } + if (!existsSync(target)) { + out.push({ + code: 'ERR_COMPONENT_NOT_FOUND', + path: pointer, + message: `${reference} names no document`, + }) + continue + } + referenced.add(realpathSync(target)) + const component = readDocument(target) + if (component !== undefined) resolved.set(node, component) + } + + checkUnreferencedComponents(itemRoot, documentPath, referenced, out) + checkConnectionOutputs(components, resolved, out) + checkInputMerge(components, resolved, out) +} + +/** Blueprint §3 — every component document in the item MUST be referenced. */ +function checkUnreferencedComponents( + itemRoot: string, + documentPath: string, + referenced: Set, + out: Diagnostic[], +): void { + const self = realpathSync(documentPath) + for (const path of yamlFiles(itemRoot)) { + const real = realpathSync(path) + if (real === self || referenced.has(real)) continue + if (child(readDocument(path), 'kind') !== 'COMPONENT') continue + out.push({ + code: 'ERR_UNREFERENCED_COMPONENT', + // Anchored at the mapping that should have named the file: a JSON Pointer + // addresses this document, and the file it complains about is not in it. + path: '/spec/components', + message: `${relative(itemRoot, path)} is referenced by no node`, + }) + } +} + +/** Blueprint §4.2 — `fromOutput` MUST name an output the component declares. */ +function checkConnectionOutputs( + components: Json | undefined, + resolved: Map, + out: Diagnostic[], +): void { + for (const node of keysOf(components)) { + const connections = child(child(components, node), 'connections') + for (const key of keysOf(connections)) { + const connection = child(connections, key) + const role = asString(child(connection, 'fromRole')) + const output = asString(child(connection, 'fromOutput')) + if (role === undefined || output === undefined) continue + + const producer = resolved.get(role) + // An unresolved producer is already ERR_UNKNOWN_ROLE or + // ERR_COMPONENT_NOT_FOUND; do not pile a second diagnostic on one cause. + if (producer === undefined) continue + + const outputs = child(child(child(producer, 'spec'), 'contract'), 'outputs') + if (!keysOf(outputs).includes(output)) { + out.push({ + code: 'ERR_UNKNOWN_OUTPUT', + path: `/spec/components/${token(node)}/connections/${token(key)}/fromOutput`, + message: `"${output}" is not an output of the component "${role}" deploys`, + }) + } + } + } +} + +/** + * Blueprint §5.2 — first-wins in lexicographic node-name order, and a *differing* + * redeclaration is an error rather than a silent discard. An identical one is + * absorbed: two components that agree on what `adminPassword` is are not in + * conflict. + * + * `ui` and `isRequired` are deliberately not compared. They describe how a value + * is asked for, not what it is. + */ +function checkInputMerge( + components: Json | undefined, + resolved: Map, + out: Diagnostic[], +): void { + const taken = new Map() + + for (const node of keysOf(components).sort()) { + const component = resolved.get(node) + if (component === undefined) continue + const inputs = child(child(child(component, 'spec'), 'contract'), 'inputs') + + for (const key of keysOf(inputs)) { + const input = child(inputs, key) + // A CONNECTION input is satisfied by a wire, never by the install form, + // so it never reaches the merge (§5.1). + if (child(input, 'suppliedBy') === 'CONNECTION') continue + + const schema = JSON.stringify(child(input, 'schema') ?? null) + const earlier = taken.get(key) + if (earlier === undefined) { + taken.set(key, { node, schema }) + continue + } + if (earlier.schema === schema) continue + out.push({ + code: 'ERR_CONFLICTING_INPUT_SCHEMA', + path: `/spec/components/${token(node)}/component`, + message: `input "${key}" is declared with a different schema by node "${earlier.node}"`, + }) + } + } +} + +// =========================================================================== +// entry point +// =========================================================================== + +/** + * Every semantic diagnostic one document produces. Ordering within the result is + * not normative — the conformance contract is that a declared diagnostic is + * among those produced, not that it is produced first. + */ +export function semanticDiagnostics( + family: Family, + document: Json, + context: SemanticContext = {}, +): Diagnostic[] { + const out: Diagnostic[] = [] + + if (family.name === 'component') { + checkImageRef(document, out) + checkProbeEndpoints(document, out) + } + if (family.name === 'blueprint') { + checkConnectionRoles(document, out) + checkCycles(document, out) + } + if (family.name === 'listing') { + checkScreenshotBasenames(document, out) + } + + const { itemRoot } = context + if (itemRoot === undefined) return out + + const documentPath = + context.documentPath ?? + join(itemRoot, family.name === 'blueprint' ? 'blueprint.yaml' : 'listing.yaml') + + if (family.name === 'blueprint' || family.name === 'listing') { + checkIdentity(family, document, itemRoot, out) + } + if (family.name === 'blueprint') { + checkGraphAgainstItem(document, itemRoot, documentPath, out) + } + if (family.name === 'listing') { + checkMediaOnDisk(document, itemRoot, out) + } + + return out +} From 920848a142269a6d2839f34a7d0d498ad921a799 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sun, 9 Aug 2026 05:21:28 +0000 Subject: [PATCH 4/6] fix(listing): correct the stale media-path description MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `$defs.ListingSpec.description` still told readers that `icon` and `screenshots[].file` are "paths relative to the listing document" — the exact wording §5 now singles out as wrong: "`media/` is a stronger rule than 'relative to the listing document'. That was this section's earlier wording, and it is not what anything enforces." PR #13 updated the two per-property descriptions and missed the containing object's, so the schema contradicted itself and the prose in one file. Refs: #9 Signed-off-by: Justin Merrell --- specifications/listing/v1/schemas/dist/listing.schema.json | 2 +- specifications/listing/v1/schemas/src/listing.schema.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/specifications/listing/v1/schemas/dist/listing.schema.json b/specifications/listing/v1/schemas/dist/listing.schema.json index cceefae..099c137 100644 --- a/specifications/listing/v1/schemas/dist/listing.schema.json +++ b/specifications/listing/v1/schemas/dist/listing.schema.json @@ -86,7 +86,7 @@ "type": "object" }, "ListingSpec": { - "description": "The ``spec`` block of a listing document — how the item is presented,\ncategorised, and discovered on the storefront.\n\nIdentity lives in ``metadata``, not here. ``icon`` and\n``screenshots[].file`` are paths relative to the listing document.", + "description": "The ``spec`` block of a listing document — how the item is presented,\ncategorised, and discovered on the storefront.\n\nIdentity lives in ``metadata``, not here. ``icon`` and\n``screenshots[].file`` are media paths, resolved inside the item root and\nrequired to begin with ``media/``.", "additionalProperties": false, "properties": { "description": { diff --git a/specifications/listing/v1/schemas/src/listing.schema.json b/specifications/listing/v1/schemas/src/listing.schema.json index 7a0861f..47f1af1 100644 --- a/specifications/listing/v1/schemas/src/listing.schema.json +++ b/specifications/listing/v1/schemas/src/listing.schema.json @@ -87,7 +87,7 @@ }, "ListingSpec": { "additionalProperties": false, - "description": "The ``spec`` block of a listing document — how the item is presented,\ncategorised, and discovered on the storefront.\n\nIdentity lives in ``metadata``, not here. ``icon`` and\n``screenshots[].file`` are paths relative to the listing document.", + "description": "The ``spec`` block of a listing document — how the item is presented,\ncategorised, and discovered on the storefront.\n\nIdentity lives in ``metadata``, not here. ``icon`` and\n``screenshots[].file`` are media paths, resolved inside the item root and\nrequired to begin with ``media/``.", "properties": { "category": { "description": "Storefront category the listing is browsed under.", From fc0f5c67ad61d63f9cba415d8ea9c06c311124e7 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sun, 9 Aug 2026 05:21:36 +0000 Subject: [PATCH 5/6] test(conformance): the item-scoped semantic cases, against ADR 0002 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Thirteen new cases covering every diagnostic code that needed a directory to exist. All are trees; none was expressible before. blueprint 003–009 take the identity, reference and merge rules one at a time — slug against the directory name, version against the sibling listing, a component document nothing references, a reference resolving to nothing, a reference escaping the item root, a `fromOutput` naming no output of the component it resolves to, and two nodes declaring one input key with differing schemas. Three deserve their reasons stated: 007 the reference is `../shared/postgres.yaml` and the document it lands on exists. A leading `../` is legal spelling — it is how a blueprint reaches a sibling directory — so only a containment check on the resolved path catches this one. 009 `api` and `db` both declare `adminPassword`, STRING against INTEGER. `api` sorts first, so `api`'s declaration stands and `db`'s is reported. The case exists because silent first-wins is defensible until you write down which schema the second component's value was validated against. 011 a three-node cycle written as queue, db, cache. semantic-002 cannot make this case: with two nodes, "smallest first" and "first declared" can coincide by luck. Here they do not, so an implementation reporting from wherever its traversal started produces the wrong walk and fails. listing 002–005 mirror the identity rules and add the two media rules that need the filesystem. 005 is the symlink case, declared in metadata rather than committed per ADR 0002 §3. blueprint 010 and listing 006 are the positive cases, and they are not padding. Each satisfies every item-scoped rule of its family at once, which makes them the regression pin for the class: a careless containment check or a merge rule that forgets to absorb an identical redeclaration rejects a well-formed item, and nothing else in the corpus would notice. `ERR_UNKNOWN_COMPONENT` remains uncovered. It is `capability` — resolving a published reference needs the catalog, and no phase a client runs may reach the network. Refs: #9 Signed-off-by: Justin Merrell --- conformance/README.md | 140 +++++++++++++----- conformance/blueprint/v1/cases.json | 60 ++++++++ .../diagnostics.json | 6 + .../metadata.json | 8 + .../tree/acme-wiki/blueprint.yaml | 16 ++ .../tree/acme-wiki/components/postgres.yaml | 15 ++ .../tree/acme-wiki/listing.yaml | 11 ++ .../diagnostics.json | 6 + .../metadata.json | 8 + .../tree/acme-wiki/blueprint.yaml | 15 ++ .../tree/acme-wiki/components/postgres.yaml | 15 ++ .../tree/acme-wiki/listing.yaml | 11 ++ .../diagnostics.json | 6 + .../metadata.json | 8 + .../tree/acme-wiki/blueprint.yaml | 15 ++ .../acme-wiki/components/postgres-legacy.yaml | 15 ++ .../tree/acme-wiki/components/postgres.yaml | 15 ++ .../tree/acme-wiki/listing.yaml | 11 ++ .../diagnostics.json | 10 ++ .../metadata.json | 8 + .../tree/acme-wiki/blueprint.yaml | 15 ++ .../tree/acme-wiki/components/mysql.yaml | 15 ++ .../tree/acme-wiki/listing.yaml | 11 ++ .../diagnostics.json | 6 + .../metadata.json | 8 + .../tree/acme-wiki/blueprint.yaml | 15 ++ .../tree/acme-wiki/listing.yaml | 11 ++ .../tree/shared/postgres.yaml | 15 ++ .../diagnostics.json | 6 + .../metadata.json | 8 + .../tree/acme-wiki/blueprint.yaml | 22 +++ .../tree/acme-wiki/components/postgres.yaml | 24 +++ .../tree/acme-wiki/components/web.yaml | 15 ++ .../tree/acme-wiki/listing.yaml | 11 ++ .../diagnostics.json | 6 + .../metadata.json | 8 + .../tree/acme-wiki/blueprint.yaml | 23 +++ .../tree/acme-wiki/components/api.yaml | 25 ++++ .../tree/acme-wiki/components/postgres.yaml | 25 ++++ .../tree/acme-wiki/listing.yaml | 11 ++ .../010-well-formed-item/metadata.json | 8 + .../tree/acme-wiki/blueprint.yaml | 27 ++++ .../tree/acme-wiki/components/postgres.yaml | 31 ++++ .../tree/acme-wiki/components/web.yaml | 25 ++++ .../tree/acme-wiki/listing.yaml | 11 ++ .../011-three-node-cycle-reporting/case.yaml | 39 +++++ .../diagnostics.json | 6 + .../metadata.json | 7 + conformance/component/v1/cases.json | 15 ++ conformance/listing/v1/cases.json | 40 +++++ .../diagnostics.json | 6 + .../metadata.json | 8 + .../tree/acme-wiki/blueprint.yaml | 12 ++ .../tree/acme-wiki/components/postgres.yaml | 10 ++ .../tree/acme-wiki/listing.yaml | 16 ++ .../tree/acme-wiki/media/icon.png | 0 .../tree/acme-wiki/media/overview.png | 0 .../diagnostics.json | 6 + .../metadata.json | 8 + .../tree/acme-wiki/blueprint.yaml | 12 ++ .../tree/acme-wiki/components/postgres.yaml | 10 ++ .../tree/acme-wiki/listing.yaml | 17 +++ .../tree/acme-wiki/media/icon.png | 0 .../tree/acme-wiki/media/overview.png | 0 .../004-media-file-absent/diagnostics.json | 6 + .../004-media-file-absent/metadata.json | 8 + .../tree/acme-wiki/blueprint.yaml | 12 ++ .../tree/acme-wiki/components/postgres.yaml | 10 ++ .../tree/acme-wiki/listing.yaml | 17 +++ .../tree/acme-wiki/media/icon.png | 0 .../diagnostics.json | 6 + .../metadata.json | 11 ++ .../tree/acme-wiki/blueprint.yaml | 12 ++ .../tree/acme-wiki/components/postgres.yaml | 10 ++ .../tree/acme-wiki/listing.yaml | 20 +++ .../tree/acme-wiki/media/overview.png | 0 .../006-well-formed-item/metadata.json | 8 + .../tree/acme-wiki/blueprint.yaml | 12 ++ .../tree/acme-wiki/components/postgres.yaml | 10 ++ .../tree/acme-wiki/listing.yaml | 17 +++ .../tree/acme-wiki/media/icon.png | 0 .../tree/acme-wiki/media/overview.png | 0 82 files changed, 1117 insertions(+), 35 deletions(-) create mode 100644 conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/diagnostics.json create mode 100644 conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/metadata.json create mode 100644 conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/tree/acme-wiki/blueprint.yaml create mode 100644 conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/tree/acme-wiki/components/postgres.yaml create mode 100644 conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/tree/acme-wiki/listing.yaml create mode 100644 conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/diagnostics.json create mode 100644 conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/metadata.json create mode 100644 conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/tree/acme-wiki/blueprint.yaml create mode 100644 conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/tree/acme-wiki/components/postgres.yaml create mode 100644 conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/tree/acme-wiki/listing.yaml create mode 100644 conformance/blueprint/v1/semantic/005-unreferenced-component-document/diagnostics.json create mode 100644 conformance/blueprint/v1/semantic/005-unreferenced-component-document/metadata.json create mode 100644 conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/blueprint.yaml create mode 100644 conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/components/postgres-legacy.yaml create mode 100644 conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/components/postgres.yaml create mode 100644 conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/listing.yaml create mode 100644 conformance/blueprint/v1/semantic/006-reference-names-no-document/diagnostics.json create mode 100644 conformance/blueprint/v1/semantic/006-reference-names-no-document/metadata.json create mode 100644 conformance/blueprint/v1/semantic/006-reference-names-no-document/tree/acme-wiki/blueprint.yaml create mode 100644 conformance/blueprint/v1/semantic/006-reference-names-no-document/tree/acme-wiki/components/mysql.yaml create mode 100644 conformance/blueprint/v1/semantic/006-reference-names-no-document/tree/acme-wiki/listing.yaml create mode 100644 conformance/blueprint/v1/semantic/007-reference-escaping-the-item/diagnostics.json create mode 100644 conformance/blueprint/v1/semantic/007-reference-escaping-the-item/metadata.json create mode 100644 conformance/blueprint/v1/semantic/007-reference-escaping-the-item/tree/acme-wiki/blueprint.yaml create mode 100644 conformance/blueprint/v1/semantic/007-reference-escaping-the-item/tree/acme-wiki/listing.yaml create mode 100644 conformance/blueprint/v1/semantic/007-reference-escaping-the-item/tree/shared/postgres.yaml create mode 100644 conformance/blueprint/v1/semantic/008-connection-names-unknown-output/diagnostics.json create mode 100644 conformance/blueprint/v1/semantic/008-connection-names-unknown-output/metadata.json create mode 100644 conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/blueprint.yaml create mode 100644 conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/components/postgres.yaml create mode 100644 conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/components/web.yaml create mode 100644 conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/listing.yaml create mode 100644 conformance/blueprint/v1/semantic/009-conflicting-input-schema/diagnostics.json create mode 100644 conformance/blueprint/v1/semantic/009-conflicting-input-schema/metadata.json create mode 100644 conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/blueprint.yaml create mode 100644 conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/components/api.yaml create mode 100644 conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/components/postgres.yaml create mode 100644 conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/listing.yaml create mode 100644 conformance/blueprint/v1/semantic/010-well-formed-item/metadata.json create mode 100644 conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/blueprint.yaml create mode 100644 conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/components/postgres.yaml create mode 100644 conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/components/web.yaml create mode 100644 conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/listing.yaml create mode 100644 conformance/blueprint/v1/semantic/011-three-node-cycle-reporting/case.yaml create mode 100644 conformance/blueprint/v1/semantic/011-three-node-cycle-reporting/diagnostics.json create mode 100644 conformance/blueprint/v1/semantic/011-three-node-cycle-reporting/metadata.json create mode 100644 conformance/listing/v1/semantic/002-slug-disagrees-with-directory/diagnostics.json create mode 100644 conformance/listing/v1/semantic/002-slug-disagrees-with-directory/metadata.json create mode 100644 conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/blueprint.yaml create mode 100644 conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/components/postgres.yaml create mode 100644 conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/listing.yaml create mode 100644 conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/media/icon.png create mode 100644 conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/media/overview.png create mode 100644 conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/diagnostics.json create mode 100644 conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/metadata.json create mode 100644 conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/blueprint.yaml create mode 100644 conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/components/postgres.yaml create mode 100644 conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/listing.yaml create mode 100644 conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/media/icon.png create mode 100644 conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/media/overview.png create mode 100644 conformance/listing/v1/semantic/004-media-file-absent/diagnostics.json create mode 100644 conformance/listing/v1/semantic/004-media-file-absent/metadata.json create mode 100644 conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/blueprint.yaml create mode 100644 conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/components/postgres.yaml create mode 100644 conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/listing.yaml create mode 100644 conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/media/icon.png create mode 100644 conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/diagnostics.json create mode 100644 conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/metadata.json create mode 100644 conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/blueprint.yaml create mode 100644 conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/components/postgres.yaml create mode 100644 conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/listing.yaml create mode 100644 conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/media/overview.png create mode 100644 conformance/listing/v1/semantic/006-well-formed-item/metadata.json create mode 100644 conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/blueprint.yaml create mode 100644 conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/components/postgres.yaml create mode 100644 conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/listing.yaml create mode 100644 conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/media/icon.png create mode 100644 conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/media/overview.png diff --git a/conformance/README.md b/conformance/README.md index 6d064dd..2a56523 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -20,10 +20,14 @@ conformance//v/ cases.json index of every case // metadata.json declared outcome - case.yaml the document under test + case.yaml the document under test, with no item root + tree/ …or the item it sits inside diagnostics.json required when expected == "fail" ``` +A case declares its subject as **exactly one** of `case.yaml` or `tree/`. See +[Case trees](#case-trees). + ## `cases.json` ```json @@ -58,6 +62,63 @@ directory without indexing it is a no-op — index entries are the contract. | `expected` | REQUIRED. `pass` or `fail`. | | `clause` | RECOMMENDED. Link to the normative clause the case exercises. Every case should trace to prose. | | `summary` | RECOMMENDED. One sentence, present tense. | +| `document` | REQUIRED for a tree case, forbidden otherwise. Path of the document under test, relative to `tree/`. | +| `symlinks` | OPTIONAL, tree cases only. Link path → link target, both verbatim. | + +## Case trees + +Some rules are about a document's *surroundings* rather than its contents: that +its slug matches the directory holding it, that a reference resolves to a file, +that a media path stays inside the item. A single `case.yaml` cannot state any +of them, so a case may instead carry a `tree/`: + +``` +conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/ + metadata.json "document": "acme-wiki/blueprint.yaml" + diagnostics.json + tree/ + acme-wiki/ <- the item root + blueprint.yaml + listing.yaml + components/postgres.yaml +``` + +The **item root** is the directory containing `document`. `tree/` is its +parent, not the item root itself — `ERR_SLUG_MISMATCH` tests the item +directory's *name*, so the tree has to contain a directory that has one. + +**`case.yaml` is not the legacy form.** It asserts that the document has **no** +item root, which [blueprint §3.1](../specifications/blueprint/v1/spec.md#item-directory) +makes a real state: a document submitted over an API arrives without a +directory, and an implementation in that position MUST NOT report any rule +measured against one. An adapter that invents an item root for a `case.yaml` is +wrong. + +**Symlinks are declared, not committed.** `ERR_PATH_ESCAPE` needs a link +resolving outside the item, and a committed one does not survive a checkout +without `core.symlinks`, is invisible in a diff, and would ship inside a release +tarball pointing outside the archive. `metadata.symlinks` names them instead: + +```json +{ "symlinks": { "acme-wiki/media/icon.png": "../../../secrets.png" } } +``` + +An adapter copies `tree/` somewhere writable, creates the links there, and runs +against the copy. The target need not exist — a dangling link resolving outside +the item root is still an escape, because containment is a property of the +resolved location rather than of the string. + +**Media files in a tree are zero bytes.** The rules they exercise are existence +and containment; nothing decodes them. A real image would make the fixture +larger without making it say more, and would invite a reader to think the +dimensions mattered — [listing §5](../specifications/listing/v1/spec.md#media) +records that they do not. + +An adapter that cannot materialise a tree SKIPs those cases. It MUST NOT report +them as passed. + +The contract is set by +[ADR 0002](../docs/adr/0002-conformance-case-trees.md). ## `diagnostics.json` @@ -91,7 +152,7 @@ identical corpus. | Phase | Enforces | Network | |---|---|---| -| `parser` | Strict YAML 1.2 — duplicate keys and aliases rejected | Never | +| `parser` | Strict YAML 1.2 — duplicate keys, anchors and aliases rejected | Never | | `structural` | The family's JSON Schema 2020-12 bundle | Never | | `semantic` | Reference resolution, path containment, dependency cycles | Never | | `capability` | Account, region, and quota checks | Server only | @@ -101,48 +162,57 @@ later-phase diagnostic before the earlier phases pass. ## Coverage status -`parser` and `structural` are covered. `semantic` is covered for every rule a -single document can express — a floating image tag, a probe naming an endpoint -that is not there, a connection naming a node that is not there, a cyclic -graph, two screenshots sharing a basename. `capability` has no cases. - -The gap is not which rules are written down; it is what a case can say. Every -remaining `semantic` rule is about a document's surroundings rather than its -contents: +`parser`, `structural` and `semantic` are covered. Every diagnostic code the +three `spec.md` files declare is exercised by at least one case, with one +exception: -| Rule | Needs | +| Code | Why it has no case | |---|---| -| `ERR_SLUG_MISMATCH` | a directory with a name | -| `ERR_VERSION_MISMATCH` | a sibling document | -| `ERR_UNREFERENCED_COMPONENT` | the item's other files | -| `ERR_COMPONENT_NOT_FOUND`, `ERR_REFERENCE_ESCAPE` | a resolvable target | -| `ERR_UNKNOWN_OUTPUT` | the referenced component document | -| `ERR_MEDIA_NOT_FOUND`, `ERR_PATH_ESCAPE` | a file on disk | - -A case is one `case.yaml`, so none of those is expressible. Extending the -fixture contract to a case tree is an ADR-gated change — GOVERNANCE.md lists -"Changing the conformance fixture contract" among the changes needing one — -and those cases land against that ADR rather than being approximated in the -meantime. - -An adapter encountering a phase it does not implement SHOULD skip the case and -report it as skipped. It MUST NOT report it as passed. +| `ERR_UNKNOWN_COMPONENT` | `capability` — resolving a published reference needs the catalog, and no phase a client runs may reach the network | + +That table is not prose anyone has to remember to update. +`task check:conformance` derives it: every `ERR_*` row in a family's own +diagnostics table must be exercised by an indexed case or appear in the +runner's `UNCOVERED` list with a reason. A code goes untested only by someone +writing down why, in a diff a reviewer sees. + +The check runs in both directions, and the second one is the reason for the +first. Before it existed, ten codes reached `main` with no fixture and CI +green — the corpus could only tell you that the cases it had were right, never +that it had the cases it needed. + +An adapter encountering a phase it does not implement, or a case shape it does +not support, SHOULD skip the case and report it as skipped. It MUST NOT report +it as passed. ## Adding a case 1. Pick the phase and the next free sequence number in that phase. -2. Create `/-/` with `metadata.json` and `case.yaml`. +2. Create `/-/` with `metadata.json`, and either a + `case.yaml` or a `tree/` plus a `document` naming the file inside it. 3. For a failing case, add `diagnostics.json`. -4. Add the entry to `cases.json`. +4. Add the entry to `cases.json`. A directory that is not indexed runs nowhere, + and `task check:conformance` reports it rather than leaving it to be assumed + green. 5. Run `task check:conformance`. A case that does not cite a `clause` will be questioned in review. Fixtures exist to pin down prose, not to freeze current implementation behaviour. -`task check:conformance` checks two separate things, and only the first of them -needs an implemented phase. Whether or not a case can be *executed* here, its -metadata is validated: the `id` leads with its phase, `cases.json` and -`metadata.json` agree on that phase, the `clause` resolves to an anchor that -exists in the cited `spec.md`, and every declared `code` appears in a -diagnostics table reachable from the family's own — at the phase that table -assigns it. A `semantic` fixture is skipped for execution but not for this. +`task check:conformance` checks three separate things, and only the first needs +an implemented phase. + +**Executing** a case runs the document through the pipeline and compares the +outcome. `tools/src/conformance.ts` implements `parser`, `structural` and +`semantic`; a `capability` case is skipped. + +**Validating** a case runs whether or not its phase does: the `id` leads with +its phase, `cases.json` and `metadata.json` agree on that phase, the case +declares exactly one of `case.yaml` or `tree/`, the `clause` resolves to an +anchor that exists in the cited `spec.md`, and every declared `code` appears in +a diagnostics table reachable from the family's own — at the phase that table +assigns it. + +**Auditing** the corpus asks the questions no individual case can: does every +declared diagnostic code have a fixture, and does every case directory on disk +appear in `cases.json`. diff --git a/conformance/blueprint/v1/cases.json b/conformance/blueprint/v1/cases.json index 74eb5da..5842b7f 100644 --- a/conformance/blueprint/v1/cases.json +++ b/conformance/blueprint/v1/cases.json @@ -7,6 +7,21 @@ "phase": "parser", "path": "parser/001-reject-duplicate-keys" }, + { + "id": "parser-002-reject-anchors-and-aliases", + "phase": "parser", + "path": "parser/002-reject-anchors-and-aliases" + }, + { + "id": "parser-003-reject-lone-anchor", + "phase": "parser", + "path": "parser/003-reject-lone-anchor" + }, + { + "id": "parser-004-malformed-yaml", + "phase": "parser", + "path": "parser/004-malformed-yaml" + }, { "id": "structural-001-minimal-valid", "phase": "structural", @@ -76,6 +91,51 @@ "id": "semantic-002-connection-cycle", "phase": "semantic", "path": "semantic/002-connection-cycle" + }, + { + "id": "semantic-003-slug-disagrees-with-directory", + "phase": "semantic", + "path": "semantic/003-slug-disagrees-with-directory" + }, + { + "id": "semantic-004-version-disagrees-with-listing", + "phase": "semantic", + "path": "semantic/004-version-disagrees-with-listing" + }, + { + "id": "semantic-005-unreferenced-component-document", + "phase": "semantic", + "path": "semantic/005-unreferenced-component-document" + }, + { + "id": "semantic-006-reference-names-no-document", + "phase": "semantic", + "path": "semantic/006-reference-names-no-document" + }, + { + "id": "semantic-007-reference-escaping-the-item", + "phase": "semantic", + "path": "semantic/007-reference-escaping-the-item" + }, + { + "id": "semantic-008-connection-names-unknown-output", + "phase": "semantic", + "path": "semantic/008-connection-names-unknown-output" + }, + { + "id": "semantic-009-conflicting-input-schema", + "phase": "semantic", + "path": "semantic/009-conflicting-input-schema" + }, + { + "id": "semantic-010-well-formed-item", + "phase": "semantic", + "path": "semantic/010-well-formed-item" + }, + { + "id": "semantic-011-three-node-cycle-reporting", + "phase": "semantic", + "path": "semantic/011-three-node-cycle-reporting" } ] } diff --git a/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/diagnostics.json b/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/diagnostics.json new file mode 100644 index 0000000..0e047dc --- /dev/null +++ b/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_SLUG_MISMATCH", + "path": "/metadata/slug" + } +] diff --git a/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/metadata.json b/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/metadata.json new file mode 100644 index 0000000..44e4474 --- /dev/null +++ b/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/metadata.json @@ -0,0 +1,8 @@ +{ + "id": "semantic-003-slug-disagrees-with-directory", + "phase": "semantic", + "expected": "fail", + "clause": "specifications/blueprint/v1/spec.md#identity", + "summary": "A blueprint whose slug is not its item directory name is rejected.", + "document": "acme-wiki/blueprint.yaml" +} diff --git a/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/tree/acme-wiki/blueprint.yaml b/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/tree/acme-wiki/blueprint.yaml new file mode 100644 index 0000000..5d3c745 --- /dev/null +++ b/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/tree/acme-wiki/blueprint.yaml @@ -0,0 +1,16 @@ +# The directory is `acme-wiki` and the document calls itself +# `acme-wiki-staging`. Nothing inside the document is wrong; the disagreement +# only exists once the document has a directory, which is what makes this a +# tree case rather than a case.yaml. +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki-staging + version: 1 +spec: + components: + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: {} + parameters: {} diff --git a/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/tree/acme-wiki/components/postgres.yaml b/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/tree/acme-wiki/components/postgres.yaml new file mode 100644 index 0000000..9508428 --- /dev/null +++ b/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/tree/acme-wiki/components/postgres.yaml @@ -0,0 +1,15 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine + endpoints: + primary: + containerPort: 5432 + protocol: TCP + visibility: PRIVATE diff --git a/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/tree/acme-wiki/listing.yaml b/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/tree/acme-wiki/listing.yaml new file mode 100644 index 0000000..da5eed4 --- /dev/null +++ b/conformance/blueprint/v1/semantic/003-slug-disagrees-with-directory/tree/acme-wiki/listing.yaml @@ -0,0 +1,11 @@ +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: BLUEPRINT + displayName: Acme Wiki + summary: A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE diff --git a/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/diagnostics.json b/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/diagnostics.json new file mode 100644 index 0000000..feb52dd --- /dev/null +++ b/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_VERSION_MISMATCH", + "path": "/metadata/version" + } +] diff --git a/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/metadata.json b/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/metadata.json new file mode 100644 index 0000000..9dbc8b3 --- /dev/null +++ b/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/metadata.json @@ -0,0 +1,8 @@ +{ + "id": "semantic-004-version-disagrees-with-listing", + "phase": "semantic", + "expected": "fail", + "clause": "specifications/blueprint/v1/spec.md#identity", + "summary": "A blueprint whose version differs from its sibling listing's is rejected.", + "document": "acme-wiki/blueprint.yaml" +} diff --git a/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/tree/acme-wiki/blueprint.yaml b/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/tree/acme-wiki/blueprint.yaml new file mode 100644 index 0000000..f5260f2 --- /dev/null +++ b/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/tree/acme-wiki/blueprint.yaml @@ -0,0 +1,15 @@ +# The graph has moved to version 2 and the storefront copy is still at 1. +# Installed, this item would deploy one thing and describe another — which is +# the whole reason the two halves are pinned to each other. +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 2 +spec: + components: + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: {} + parameters: {} diff --git a/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/tree/acme-wiki/components/postgres.yaml b/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/tree/acme-wiki/components/postgres.yaml new file mode 100644 index 0000000..9508428 --- /dev/null +++ b/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/tree/acme-wiki/components/postgres.yaml @@ -0,0 +1,15 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine + endpoints: + primary: + containerPort: 5432 + protocol: TCP + visibility: PRIVATE diff --git a/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/tree/acme-wiki/listing.yaml b/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/tree/acme-wiki/listing.yaml new file mode 100644 index 0000000..da5eed4 --- /dev/null +++ b/conformance/blueprint/v1/semantic/004-version-disagrees-with-listing/tree/acme-wiki/listing.yaml @@ -0,0 +1,11 @@ +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: BLUEPRINT + displayName: Acme Wiki + summary: A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE diff --git a/conformance/blueprint/v1/semantic/005-unreferenced-component-document/diagnostics.json b/conformance/blueprint/v1/semantic/005-unreferenced-component-document/diagnostics.json new file mode 100644 index 0000000..a1a7404 --- /dev/null +++ b/conformance/blueprint/v1/semantic/005-unreferenced-component-document/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_UNREFERENCED_COMPONENT", + "path": "/spec/components" + } +] diff --git a/conformance/blueprint/v1/semantic/005-unreferenced-component-document/metadata.json b/conformance/blueprint/v1/semantic/005-unreferenced-component-document/metadata.json new file mode 100644 index 0000000..79d2f6d --- /dev/null +++ b/conformance/blueprint/v1/semantic/005-unreferenced-component-document/metadata.json @@ -0,0 +1,8 @@ +{ + "id": "semantic-005-unreferenced-component-document", + "phase": "semantic", + "expected": "fail", + "clause": "specifications/blueprint/v1/spec.md#identity", + "summary": "A component document in the item that no node references is rejected.", + "document": "acme-wiki/blueprint.yaml" +} diff --git a/conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/blueprint.yaml b/conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/blueprint.yaml new file mode 100644 index 0000000..57b851b --- /dev/null +++ b/conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/blueprint.yaml @@ -0,0 +1,15 @@ +# `components/postgres-legacy.yaml` is last release's document, left beside the +# one in use. Nothing in the directory says which is live, and a reader working +# out what the item deploys has to read the graph to find out. +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 1 +spec: + components: + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: {} + parameters: {} diff --git a/conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/components/postgres-legacy.yaml b/conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/components/postgres-legacy.yaml new file mode 100644 index 0000000..9508428 --- /dev/null +++ b/conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/components/postgres-legacy.yaml @@ -0,0 +1,15 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine + endpoints: + primary: + containerPort: 5432 + protocol: TCP + visibility: PRIVATE diff --git a/conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/components/postgres.yaml b/conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/components/postgres.yaml new file mode 100644 index 0000000..9508428 --- /dev/null +++ b/conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/components/postgres.yaml @@ -0,0 +1,15 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine + endpoints: + primary: + containerPort: 5432 + protocol: TCP + visibility: PRIVATE diff --git a/conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/listing.yaml b/conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/listing.yaml new file mode 100644 index 0000000..da5eed4 --- /dev/null +++ b/conformance/blueprint/v1/semantic/005-unreferenced-component-document/tree/acme-wiki/listing.yaml @@ -0,0 +1,11 @@ +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: BLUEPRINT + displayName: Acme Wiki + summary: A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE diff --git a/conformance/blueprint/v1/semantic/006-reference-names-no-document/diagnostics.json b/conformance/blueprint/v1/semantic/006-reference-names-no-document/diagnostics.json new file mode 100644 index 0000000..c6e69e2 --- /dev/null +++ b/conformance/blueprint/v1/semantic/006-reference-names-no-document/diagnostics.json @@ -0,0 +1,10 @@ +[ + { + "code": "ERR_COMPONENT_NOT_FOUND", + "path": "/spec/components/db/component" + }, + { + "code": "ERR_UNREFERENCED_COMPONENT", + "path": "/spec/components" + } +] diff --git a/conformance/blueprint/v1/semantic/006-reference-names-no-document/metadata.json b/conformance/blueprint/v1/semantic/006-reference-names-no-document/metadata.json new file mode 100644 index 0000000..ae2d333 --- /dev/null +++ b/conformance/blueprint/v1/semantic/006-reference-names-no-document/metadata.json @@ -0,0 +1,8 @@ +{ + "id": "semantic-006-reference-names-no-document", + "phase": "semantic", + "expected": "fail", + "clause": "specifications/blueprint/v1/spec.md#component-reference", + "summary": "A repo-local reference resolving to no document is rejected.", + "document": "acme-wiki/blueprint.yaml" +} diff --git a/conformance/blueprint/v1/semantic/006-reference-names-no-document/tree/acme-wiki/blueprint.yaml b/conformance/blueprint/v1/semantic/006-reference-names-no-document/tree/acme-wiki/blueprint.yaml new file mode 100644 index 0000000..f3e8563 --- /dev/null +++ b/conformance/blueprint/v1/semantic/006-reference-names-no-document/tree/acme-wiki/blueprint.yaml @@ -0,0 +1,15 @@ +# The reference is well-formed and resolves to nothing: the item ships +# `mysql.yaml` and the graph asks for `postgres.yaml`. A typo in a path is +# indistinguishable from a missing file until something looks on disk. +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 1 +spec: + components: + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: {} + parameters: {} diff --git a/conformance/blueprint/v1/semantic/006-reference-names-no-document/tree/acme-wiki/components/mysql.yaml b/conformance/blueprint/v1/semantic/006-reference-names-no-document/tree/acme-wiki/components/mysql.yaml new file mode 100644 index 0000000..9508428 --- /dev/null +++ b/conformance/blueprint/v1/semantic/006-reference-names-no-document/tree/acme-wiki/components/mysql.yaml @@ -0,0 +1,15 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine + endpoints: + primary: + containerPort: 5432 + protocol: TCP + visibility: PRIVATE diff --git a/conformance/blueprint/v1/semantic/006-reference-names-no-document/tree/acme-wiki/listing.yaml b/conformance/blueprint/v1/semantic/006-reference-names-no-document/tree/acme-wiki/listing.yaml new file mode 100644 index 0000000..da5eed4 --- /dev/null +++ b/conformance/blueprint/v1/semantic/006-reference-names-no-document/tree/acme-wiki/listing.yaml @@ -0,0 +1,11 @@ +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: BLUEPRINT + displayName: Acme Wiki + summary: A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE diff --git a/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/diagnostics.json b/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/diagnostics.json new file mode 100644 index 0000000..0072eb5 --- /dev/null +++ b/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_REFERENCE_ESCAPE", + "path": "/spec/components/db/component" + } +] diff --git a/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/metadata.json b/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/metadata.json new file mode 100644 index 0000000..cdfbc69 --- /dev/null +++ b/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/metadata.json @@ -0,0 +1,8 @@ +{ + "id": "semantic-007-reference-escaping-the-item", + "phase": "semantic", + "expected": "fail", + "clause": "specifications/blueprint/v1/spec.md#component-reference", + "summary": "A repo-local reference resolving outside the item root is rejected.", + "document": "acme-wiki/blueprint.yaml" +} diff --git a/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/tree/acme-wiki/blueprint.yaml b/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/tree/acme-wiki/blueprint.yaml new file mode 100644 index 0000000..599da5c --- /dev/null +++ b/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/tree/acme-wiki/blueprint.yaml @@ -0,0 +1,15 @@ +# A leading `../` is legal spelling — it is how a blueprint reaches a sibling +# directory — and this one reaches past the item root. The document it lands on +# exists, so nothing but a containment check on the resolved path catches it. +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 1 +spec: + components: + db: + component: ../shared/postgres.yaml + size: general.standard.small + connections: {} + parameters: {} diff --git a/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/tree/acme-wiki/listing.yaml b/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/tree/acme-wiki/listing.yaml new file mode 100644 index 0000000..da5eed4 --- /dev/null +++ b/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/tree/acme-wiki/listing.yaml @@ -0,0 +1,11 @@ +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: BLUEPRINT + displayName: Acme Wiki + summary: A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE diff --git a/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/tree/shared/postgres.yaml b/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/tree/shared/postgres.yaml new file mode 100644 index 0000000..9508428 --- /dev/null +++ b/conformance/blueprint/v1/semantic/007-reference-escaping-the-item/tree/shared/postgres.yaml @@ -0,0 +1,15 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine + endpoints: + primary: + containerPort: 5432 + protocol: TCP + visibility: PRIVATE diff --git a/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/diagnostics.json b/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/diagnostics.json new file mode 100644 index 0000000..21e87d9 --- /dev/null +++ b/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_UNKNOWN_OUTPUT", + "path": "/spec/components/web/connections/DATABASE_URL/fromOutput" + } +] diff --git a/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/metadata.json b/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/metadata.json new file mode 100644 index 0000000..a091524 --- /dev/null +++ b/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/metadata.json @@ -0,0 +1,8 @@ +{ + "id": "semantic-008-connection-names-unknown-output", + "phase": "semantic", + "expected": "fail", + "clause": "specifications/blueprint/v1/spec.md#connections", + "summary": "A connection naming an output the producing component does not declare is rejected.", + "document": "acme-wiki/blueprint.yaml" +} diff --git a/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/blueprint.yaml b/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/blueprint.yaml new file mode 100644 index 0000000..996a1f1 --- /dev/null +++ b/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/blueprint.yaml @@ -0,0 +1,22 @@ +# `db` publishes `connectionString`; `web` asks for `connectionUri`. Both node +# names resolve, so ERR_UNKNOWN_ROLE does not fire — only reading the component +# document `db` deploys settles it, which is what makes this rule item-scoped. +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 1 +spec: + components: + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: {} + web: + component: ./components/web.yaml + size: general.standard.small + connections: + DATABASE_URL: + fromRole: db + fromOutput: connectionUri + parameters: {} diff --git a/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/components/postgres.yaml b/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/components/postgres.yaml new file mode 100644 index 0000000..ac4036e --- /dev/null +++ b/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/components/postgres.yaml @@ -0,0 +1,24 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine + endpoints: + primary: + containerPort: 5432 + protocol: TCP + visibility: PRIVATE + contract: + inputs: {} + outputs: + connectionString: + schema: + type: STRING + isSensitive: true + valueFrom: DERIVED + value: null diff --git a/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/components/web.yaml b/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/components/web.yaml new file mode 100644 index 0000000..9508428 --- /dev/null +++ b/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/components/web.yaml @@ -0,0 +1,15 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine + endpoints: + primary: + containerPort: 5432 + protocol: TCP + visibility: PRIVATE diff --git a/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/listing.yaml b/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/listing.yaml new file mode 100644 index 0000000..da5eed4 --- /dev/null +++ b/conformance/blueprint/v1/semantic/008-connection-names-unknown-output/tree/acme-wiki/listing.yaml @@ -0,0 +1,11 @@ +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: BLUEPRINT + displayName: Acme Wiki + summary: A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE diff --git a/conformance/blueprint/v1/semantic/009-conflicting-input-schema/diagnostics.json b/conformance/blueprint/v1/semantic/009-conflicting-input-schema/diagnostics.json new file mode 100644 index 0000000..23cd313 --- /dev/null +++ b/conformance/blueprint/v1/semantic/009-conflicting-input-schema/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_CONFLICTING_INPUT_SCHEMA", + "path": "/spec/components/db/component" + } +] diff --git a/conformance/blueprint/v1/semantic/009-conflicting-input-schema/metadata.json b/conformance/blueprint/v1/semantic/009-conflicting-input-schema/metadata.json new file mode 100644 index 0000000..8fe7376 --- /dev/null +++ b/conformance/blueprint/v1/semantic/009-conflicting-input-schema/metadata.json @@ -0,0 +1,8 @@ +{ + "id": "semantic-009-conflicting-input-schema", + "phase": "semantic", + "expected": "fail", + "clause": "specifications/blueprint/v1/spec.md#merge", + "summary": "Two nodes declaring one input key with differing schemas are rejected.", + "document": "acme-wiki/blueprint.yaml" +} diff --git a/conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/blueprint.yaml b/conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/blueprint.yaml new file mode 100644 index 0000000..32e9375 --- /dev/null +++ b/conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/blueprint.yaml @@ -0,0 +1,23 @@ +# Both components declare `adminPassword`, and they disagree about what it is: +# a STRING for `api`, an INTEGER for `db`. Silent first-wins would hand `db` a +# value validated against `api`'s rules and fail at deploy time, a long way from +# either document. +# +# `api` sorts before `db`, so `api`'s declaration is the one that stands and +# `db`'s is the one reported. +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 1 +spec: + components: + api: + component: ./components/api.yaml + size: general.standard.small + connections: {} + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: {} + parameters: {} diff --git a/conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/components/api.yaml b/conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/components/api.yaml new file mode 100644 index 0000000..6167d2d --- /dev/null +++ b/conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/components/api.yaml @@ -0,0 +1,25 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine + endpoints: + primary: + containerPort: 5432 + protocol: TCP + visibility: PRIVATE + contract: + inputs: + adminPassword: + schema: + type: STRING + isSensitive: true + suppliedBy: USER + ui: + label: Administrator password + outputs: {} diff --git a/conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/components/postgres.yaml b/conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/components/postgres.yaml new file mode 100644 index 0000000..28b0a7a --- /dev/null +++ b/conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/components/postgres.yaml @@ -0,0 +1,25 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine + endpoints: + primary: + containerPort: 5432 + protocol: TCP + visibility: PRIVATE + contract: + inputs: + adminPassword: + schema: + type: INTEGER + isSensitive: true + suppliedBy: USER + ui: + label: Administrator password + outputs: {} diff --git a/conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/listing.yaml b/conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/listing.yaml new file mode 100644 index 0000000..da5eed4 --- /dev/null +++ b/conformance/blueprint/v1/semantic/009-conflicting-input-schema/tree/acme-wiki/listing.yaml @@ -0,0 +1,11 @@ +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: BLUEPRINT + displayName: Acme Wiki + summary: A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE diff --git a/conformance/blueprint/v1/semantic/010-well-formed-item/metadata.json b/conformance/blueprint/v1/semantic/010-well-formed-item/metadata.json new file mode 100644 index 0000000..558f21d --- /dev/null +++ b/conformance/blueprint/v1/semantic/010-well-formed-item/metadata.json @@ -0,0 +1,8 @@ +{ + "id": "semantic-010-well-formed-item", + "phase": "semantic", + "expected": "pass", + "clause": "specifications/blueprint/v1/spec.md#identity", + "summary": "An item satisfying every identity, reference and merge rule validates.", + "document": "acme-wiki/blueprint.yaml" +} diff --git a/conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/blueprint.yaml b/conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/blueprint.yaml new file mode 100644 index 0000000..c57bd49 --- /dev/null +++ b/conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/blueprint.yaml @@ -0,0 +1,27 @@ +# Every item-scoped rule satisfied at once, which is what makes this the +# regression pin: a careless implementation of any one of them rejects a +# well-formed item, and nothing else in the corpus would notice. +# +# The slug is the directory name, the version matches the listing, both +# component documents are referenced, both references resolve inside the item, +# the connection names an output `db` really publishes, and the two components +# declare `adminPassword` identically — which §5.2 absorbs rather than reports. +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 1 +spec: + components: + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: {} + web: + component: ./components/web.yaml + size: general.standard.small + connections: + DATABASE_URL: + fromRole: db + fromOutput: connectionString + parameters: {} diff --git a/conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/components/postgres.yaml b/conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/components/postgres.yaml new file mode 100644 index 0000000..3102506 --- /dev/null +++ b/conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/components/postgres.yaml @@ -0,0 +1,31 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine + endpoints: + primary: + containerPort: 5432 + protocol: TCP + visibility: PRIVATE + contract: + inputs: + adminPassword: + schema: + type: STRING + isSensitive: true + suppliedBy: USER + ui: + label: Administrator password + outputs: + connectionString: + schema: + type: STRING + isSensitive: true + valueFrom: DERIVED + value: null diff --git a/conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/components/web.yaml b/conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/components/web.yaml new file mode 100644 index 0000000..6167d2d --- /dev/null +++ b/conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/components/web.yaml @@ -0,0 +1,25 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine + endpoints: + primary: + containerPort: 5432 + protocol: TCP + visibility: PRIVATE + contract: + inputs: + adminPassword: + schema: + type: STRING + isSensitive: true + suppliedBy: USER + ui: + label: Administrator password + outputs: {} diff --git a/conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/listing.yaml b/conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/listing.yaml new file mode 100644 index 0000000..da5eed4 --- /dev/null +++ b/conformance/blueprint/v1/semantic/010-well-formed-item/tree/acme-wiki/listing.yaml @@ -0,0 +1,11 @@ +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: BLUEPRINT + displayName: Acme Wiki + summary: A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE diff --git a/conformance/blueprint/v1/semantic/011-three-node-cycle-reporting/case.yaml b/conformance/blueprint/v1/semantic/011-three-node-cycle-reporting/case.yaml new file mode 100644 index 0000000..50285eb --- /dev/null +++ b/conformance/blueprint/v1/semantic/011-three-node-cycle-reporting/case.yaml @@ -0,0 +1,39 @@ +# A three-node cycle written in an order that is not its reporting order. The +# nodes appear as queue, db, cache and the walk is required to begin at `cache`, +# the lexicographically smallest node in the cycle. +# +# This is the case semantic-002 cannot make: with two nodes, "smallest first" +# and "first declared" can coincide by luck. Here they do not, so an +# implementation that reports from wherever its traversal happened to start +# produces `queue -> db -> cache -> queue` and fails — which is the point. +# The node names in a diagnostic are comparable across implementations only +# because this rule fixes them. +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: mutual-discovery + version: 1 +spec: + components: + queue: + component: ./components/queue.yaml + size: general.standard.small + connections: + DB_URL: + fromRole: db + fromOutput: privateAddress + db: + component: ./components/db.yaml + size: general.standard.small + connections: + CACHE_URL: + fromRole: cache + fromOutput: privateAddress + cache: + component: ./components/cache.yaml + size: general.standard.small + connections: + QUEUE_URL: + fromRole: queue + fromOutput: privateAddress + parameters: {} diff --git a/conformance/blueprint/v1/semantic/011-three-node-cycle-reporting/diagnostics.json b/conformance/blueprint/v1/semantic/011-three-node-cycle-reporting/diagnostics.json new file mode 100644 index 0000000..8c7de87 --- /dev/null +++ b/conformance/blueprint/v1/semantic/011-three-node-cycle-reporting/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_DEPENDENCY_CYCLE", + "path": "/spec/components/cache/connections" + } +] diff --git a/conformance/blueprint/v1/semantic/011-three-node-cycle-reporting/metadata.json b/conformance/blueprint/v1/semantic/011-three-node-cycle-reporting/metadata.json new file mode 100644 index 0000000..1f44bfc --- /dev/null +++ b/conformance/blueprint/v1/semantic/011-three-node-cycle-reporting/metadata.json @@ -0,0 +1,7 @@ +{ + "id": "semantic-011-three-node-cycle-reporting", + "phase": "semantic", + "expected": "fail", + "clause": "specifications/blueprint/v1/spec.md#connections", + "summary": "A cycle is reported from the lexicographically smallest node, not the first declared." +} diff --git a/conformance/component/v1/cases.json b/conformance/component/v1/cases.json index 95d57e1..0493d8d 100644 --- a/conformance/component/v1/cases.json +++ b/conformance/component/v1/cases.json @@ -7,6 +7,21 @@ "phase": "parser", "path": "parser/001-reject-duplicate-keys" }, + { + "id": "parser-002-reject-anchors-and-aliases", + "phase": "parser", + "path": "parser/002-reject-anchors-and-aliases" + }, + { + "id": "parser-003-reject-lone-anchor", + "phase": "parser", + "path": "parser/003-reject-lone-anchor" + }, + { + "id": "parser-004-malformed-yaml", + "phase": "parser", + "path": "parser/004-malformed-yaml" + }, { "id": "structural-001-minimal-valid", "phase": "structural", diff --git a/conformance/listing/v1/cases.json b/conformance/listing/v1/cases.json index b61df42..91764e5 100644 --- a/conformance/listing/v1/cases.json +++ b/conformance/listing/v1/cases.json @@ -7,6 +7,21 @@ "phase": "parser", "path": "parser/001-reject-duplicate-keys" }, + { + "id": "parser-002-reject-anchors-and-aliases", + "phase": "parser", + "path": "parser/002-reject-anchors-and-aliases" + }, + { + "id": "parser-003-reject-lone-anchor", + "phase": "parser", + "path": "parser/003-reject-lone-anchor" + }, + { + "id": "parser-004-malformed-yaml", + "phase": "parser", + "path": "parser/004-malformed-yaml" + }, { "id": "structural-001-minimal-valid", "phase": "structural", @@ -61,6 +76,31 @@ "id": "semantic-001-duplicate-screenshot-basename", "phase": "semantic", "path": "semantic/001-duplicate-screenshot-basename" + }, + { + "id": "semantic-002-slug-disagrees-with-directory", + "phase": "semantic", + "path": "semantic/002-slug-disagrees-with-directory" + }, + { + "id": "semantic-003-version-disagrees-with-blueprint", + "phase": "semantic", + "path": "semantic/003-version-disagrees-with-blueprint" + }, + { + "id": "semantic-004-media-file-absent", + "phase": "semantic", + "path": "semantic/004-media-file-absent" + }, + { + "id": "semantic-005-media-symlink-escaping-the-item", + "phase": "semantic", + "path": "semantic/005-media-symlink-escaping-the-item" + }, + { + "id": "semantic-006-well-formed-item", + "phase": "semantic", + "path": "semantic/006-well-formed-item" } ] } diff --git a/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/diagnostics.json b/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/diagnostics.json new file mode 100644 index 0000000..0e047dc --- /dev/null +++ b/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_SLUG_MISMATCH", + "path": "/metadata/slug" + } +] diff --git a/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/metadata.json b/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/metadata.json new file mode 100644 index 0000000..546a9b2 --- /dev/null +++ b/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/metadata.json @@ -0,0 +1,8 @@ +{ + "id": "semantic-002-slug-disagrees-with-directory", + "phase": "semantic", + "expected": "fail", + "clause": "specifications/listing/v1/spec.md#identity", + "summary": "A listing whose slug is not its item directory name is rejected.", + "document": "acme-wiki/listing.yaml" +} diff --git a/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/blueprint.yaml b/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/blueprint.yaml new file mode 100644 index 0000000..1599caf --- /dev/null +++ b/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/blueprint.yaml @@ -0,0 +1,12 @@ +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 1 +spec: + components: + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: {} + parameters: {} diff --git a/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/components/postgres.yaml b/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/components/postgres.yaml new file mode 100644 index 0000000..9a8c959 --- /dev/null +++ b/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/components/postgres.yaml @@ -0,0 +1,10 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine diff --git a/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/listing.yaml b/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/listing.yaml new file mode 100644 index 0000000..a260ae3 --- /dev/null +++ b/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/listing.yaml @@ -0,0 +1,16 @@ +# Listing §3 mirrors blueprint §3 and measures against the same item root. The +# directory is `acme-wiki`; this document calls itself something else. +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki-staging + version: 1 +spec: + listingKind: BLUEPRINT + displayName: Acme Wiki + summary: A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE + icon: media/icon.png + screenshots: + - file: media/overview.png diff --git a/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/media/icon.png b/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/media/icon.png new file mode 100644 index 0000000..e69de29 diff --git a/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/media/overview.png b/conformance/listing/v1/semantic/002-slug-disagrees-with-directory/tree/acme-wiki/media/overview.png new file mode 100644 index 0000000..e69de29 diff --git a/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/diagnostics.json b/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/diagnostics.json new file mode 100644 index 0000000..feb52dd --- /dev/null +++ b/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_VERSION_MISMATCH", + "path": "/metadata/version" + } +] diff --git a/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/metadata.json b/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/metadata.json new file mode 100644 index 0000000..f1523b0 --- /dev/null +++ b/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/metadata.json @@ -0,0 +1,8 @@ +{ + "id": "semantic-003-version-disagrees-with-blueprint", + "phase": "semantic", + "expected": "fail", + "clause": "specifications/listing/v1/spec.md#identity", + "summary": "A listing whose version differs from its sibling blueprint's is rejected.", + "document": "acme-wiki/listing.yaml" +} diff --git a/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/blueprint.yaml b/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/blueprint.yaml new file mode 100644 index 0000000..c844e77 --- /dev/null +++ b/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/blueprint.yaml @@ -0,0 +1,12 @@ +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 2 +spec: + components: + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: {} + parameters: {} diff --git a/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/components/postgres.yaml b/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/components/postgres.yaml new file mode 100644 index 0000000..9a8c959 --- /dev/null +++ b/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/components/postgres.yaml @@ -0,0 +1,10 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine diff --git a/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/listing.yaml b/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/listing.yaml new file mode 100644 index 0000000..2a393f4 --- /dev/null +++ b/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/listing.yaml @@ -0,0 +1,17 @@ +# The storefront copy has been left behind: this release's graph, last +# release's description. Read one and install the other and they are not the +# same item. +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: BLUEPRINT + displayName: Acme Wiki + summary: A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE + icon: media/icon.png + screenshots: + - file: media/overview.png diff --git a/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/media/icon.png b/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/media/icon.png new file mode 100644 index 0000000..e69de29 diff --git a/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/media/overview.png b/conformance/listing/v1/semantic/003-version-disagrees-with-blueprint/tree/acme-wiki/media/overview.png new file mode 100644 index 0000000..e69de29 diff --git a/conformance/listing/v1/semantic/004-media-file-absent/diagnostics.json b/conformance/listing/v1/semantic/004-media-file-absent/diagnostics.json new file mode 100644 index 0000000..a079a70 --- /dev/null +++ b/conformance/listing/v1/semantic/004-media-file-absent/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_MEDIA_NOT_FOUND", + "path": "/spec/screenshots/0/file" + } +] diff --git a/conformance/listing/v1/semantic/004-media-file-absent/metadata.json b/conformance/listing/v1/semantic/004-media-file-absent/metadata.json new file mode 100644 index 0000000..026e653 --- /dev/null +++ b/conformance/listing/v1/semantic/004-media-file-absent/metadata.json @@ -0,0 +1,8 @@ +{ + "id": "semantic-004-media-file-absent", + "phase": "semantic", + "expected": "fail", + "clause": "specifications/listing/v1/spec.md#media", + "summary": "A media path naming a file the item does not ship is rejected.", + "document": "acme-wiki/listing.yaml" +} diff --git a/conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/blueprint.yaml b/conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/blueprint.yaml new file mode 100644 index 0000000..1599caf --- /dev/null +++ b/conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/blueprint.yaml @@ -0,0 +1,12 @@ +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 1 +spec: + components: + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: {} + parameters: {} diff --git a/conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/components/postgres.yaml b/conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/components/postgres.yaml new file mode 100644 index 0000000..9a8c959 --- /dev/null +++ b/conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/components/postgres.yaml @@ -0,0 +1,10 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine diff --git a/conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/listing.yaml b/conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/listing.yaml new file mode 100644 index 0000000..4025a50 --- /dev/null +++ b/conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/listing.yaml @@ -0,0 +1,17 @@ +# `media/overview.png` satisfies the grammar and is not there. The structural +# pattern can say a path is well-formed; only the filesystem can say it points +# at something. +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: BLUEPRINT + displayName: Acme Wiki + summary: A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE + icon: media/icon.png + screenshots: + - file: media/overview.png diff --git a/conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/media/icon.png b/conformance/listing/v1/semantic/004-media-file-absent/tree/acme-wiki/media/icon.png new file mode 100644 index 0000000..e69de29 diff --git a/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/diagnostics.json b/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/diagnostics.json new file mode 100644 index 0000000..017f4a8 --- /dev/null +++ b/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/diagnostics.json @@ -0,0 +1,6 @@ +[ + { + "code": "ERR_PATH_ESCAPE", + "path": "/spec/icon" + } +] diff --git a/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/metadata.json b/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/metadata.json new file mode 100644 index 0000000..d31d2f4 --- /dev/null +++ b/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/metadata.json @@ -0,0 +1,11 @@ +{ + "id": "semantic-005-media-symlink-escaping-the-item", + "phase": "semantic", + "expected": "fail", + "clause": "specifications/listing/v1/spec.md#media", + "summary": "A media path resolving outside the item root through a symlink is rejected.", + "document": "acme-wiki/listing.yaml", + "symlinks": { + "acme-wiki/media/icon.png": "../../../secrets.png" + } +} diff --git a/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/blueprint.yaml b/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/blueprint.yaml new file mode 100644 index 0000000..1599caf --- /dev/null +++ b/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/blueprint.yaml @@ -0,0 +1,12 @@ +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 1 +spec: + components: + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: {} + parameters: {} diff --git a/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/components/postgres.yaml b/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/components/postgres.yaml new file mode 100644 index 0000000..9a8c959 --- /dev/null +++ b/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/components/postgres.yaml @@ -0,0 +1,10 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine diff --git a/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/listing.yaml b/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/listing.yaml new file mode 100644 index 0000000..ea913d5 --- /dev/null +++ b/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/listing.yaml @@ -0,0 +1,20 @@ +# `media/icon.png` is a legal spelling — §5's grammar makes `..` unspellable, +# so no path escapes by traversal any more. It escapes by symlink instead, which +# is why ERR_PATH_ESCAPE outlives the grammar: containment is a property of the +# resolved location, not of the string. +# +# The link is declared in metadata.json rather than committed. See ADR 0002 §3. +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: BLUEPRINT + displayName: Acme Wiki + summary: A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE + icon: media/icon.png + screenshots: + - file: media/overview.png diff --git a/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/media/overview.png b/conformance/listing/v1/semantic/005-media-symlink-escaping-the-item/tree/acme-wiki/media/overview.png new file mode 100644 index 0000000..e69de29 diff --git a/conformance/listing/v1/semantic/006-well-formed-item/metadata.json b/conformance/listing/v1/semantic/006-well-formed-item/metadata.json new file mode 100644 index 0000000..153c59f --- /dev/null +++ b/conformance/listing/v1/semantic/006-well-formed-item/metadata.json @@ -0,0 +1,8 @@ +{ + "id": "semantic-006-well-formed-item", + "phase": "semantic", + "expected": "pass", + "clause": "specifications/listing/v1/spec.md#identity", + "summary": "A listing agreeing with its directory, its sibling and its media validates.", + "document": "acme-wiki/listing.yaml" +} diff --git a/conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/blueprint.yaml b/conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/blueprint.yaml new file mode 100644 index 0000000..1599caf --- /dev/null +++ b/conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/blueprint.yaml @@ -0,0 +1,12 @@ +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: acme-wiki + version: 1 +spec: + components: + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: {} + parameters: {} diff --git a/conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/components/postgres.yaml b/conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/components/postgres.yaml new file mode 100644 index 0000000..9a8c959 --- /dev/null +++ b/conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/components/postgres.yaml @@ -0,0 +1,10 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine diff --git a/conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/listing.yaml b/conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/listing.yaml new file mode 100644 index 0000000..f3cf82a --- /dev/null +++ b/conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/listing.yaml @@ -0,0 +1,17 @@ +# The regression pin for the listing's item-scoped rules: slug equal to the +# directory, version equal to the sibling blueprint's, and both media paths +# resolving to files inside the item. A careless containment check rejects this. +specVersion: v1 +kind: LISTING +metadata: + slug: acme-wiki + version: 1 +spec: + listingKind: BLUEPRINT + displayName: Acme Wiki + summary: A wiki backed by PostgreSQL + category: PRODUCTIVITY + lifecycleStage: STABLE + icon: media/icon.png + screenshots: + - file: media/overview.png diff --git a/conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/media/icon.png b/conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/media/icon.png new file mode 100644 index 0000000..e69de29 diff --git a/conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/media/overview.png b/conformance/listing/v1/semantic/006-well-formed-item/tree/acme-wiki/media/overview.png new file mode 100644 index 0000000..e69de29 From 505d9b1d97cfc0aee3843b271cb3b7622aa966cf Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sun, 9 Aug 2026 05:22:03 +0000 Subject: [PATCH 6/6] docs(conformance): record coverage as a check rather than a claim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The runner enforced fixture → spec: every code a case declares must exist in a reachable diagnostics table, at the right phase. Nothing enforced spec → fixture, which is how ten codes reached `main` untested with CI green. Two checks close it. Every `ERR_*` row in a family's own diagnostics table must be exercised by an indexed case or appear in `UNCOVERED` with a reason — after this branch that list holds one entry. And every case directory on disk must be named by `cases.json`, since the runner reads the index rather than walking the tree: an unindexed directory is not a failing fixture, it is an invisible one. Both were verified by injecting a violation and watching the check fail. The README's "Coverage status" described a limitation ADR 0002 removes, so it is rewritten around what is now true: three phases covered, one code uncovered, and a check that derives the list instead of a paragraph someone has to remember to update. It also documents the `tree/` form, `document`, and `symlinks`. CONTRIBUTING gains the obligation this creates: adding a diagnostic code to a spec.md means adding a case for it. Refs: #9 Signed-off-by: Justin Merrell --- CONTRIBUTING.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 86203b1..626f2b4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -47,6 +47,15 @@ mkdir -p conformance/component/v1/structural/010-my-new-rule task check ``` +A fixture is a `case.yaml` when the rule is decided by reading one document, +and a `tree/` when it is decided by reading the item the document sits in — +a slug against its directory, a reference against a file. See +[conformance/README.md](conformance/README.md#case-trees). + +Adding a diagnostic code to a `spec.md` obliges you to add a case for it. +`check:conformance` fails otherwise, and the only way out is an entry in the +runner's `UNCOVERED` list saying why the code cannot be exercised. + `task check` runs, in order: | Step | What it enforces | @@ -56,7 +65,7 @@ task check | `check:schema` | Every `src/` module is valid JSON Schema 2020-12; `$id`s are unique and canonical; no remote `$ref` | | `check:drift` | The committed `dist/` bundle matches a fresh compile of `src/` | | `check:examples` | Every file in `examples/` validates against its family's bundle | -| `check:conformance` | Every conformance case produces its declared outcome | +| `check:conformance` | Every conformance case produces its declared outcome; every declared diagnostic code has a case; every case directory is indexed | | `check:shell` | ShellCheck over `.devcontainer/scripts` | | `check:workflow` | actionlint over `.github/workflows` |