Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 31 additions & 4 deletions conformance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,10 +101,29 @@ later-phase diagnostic before the earlier phases pass.

## Coverage status

Cases exist today for the `parser` and `structural` phases only. `semantic` and
`capability` cases land once the corresponding rules are written into `spec.md`
— a fixture without a normative clause to cite is an assertion about an
implementation, not about the specification.
`parser` and `structural` are covered. `semantic` is covered for every rule a
single document can express — a floating image tag, a probe naming an endpoint
that is not there, a connection naming a node that is not there, a cyclic
graph, two screenshots sharing a basename. `capability` has no cases.

The gap is not which rules are written down; it is what a case can say. Every
remaining `semantic` rule is about a document's surroundings rather than its
contents:

| Rule | Needs |
|---|---|
| `ERR_SLUG_MISMATCH` | a directory with a name |
| `ERR_VERSION_MISMATCH` | a sibling document |
| `ERR_UNREFERENCED_COMPONENT` | the item's other files |
| `ERR_COMPONENT_NOT_FOUND`, `ERR_REFERENCE_ESCAPE` | a resolvable target |
| `ERR_UNKNOWN_OUTPUT` | the referenced component document |
| `ERR_MEDIA_NOT_FOUND`, `ERR_PATH_ESCAPE` | a file on disk |

A case is one `case.yaml`, so none of those is expressible. Extending the
fixture contract to a case tree is an ADR-gated change — GOVERNANCE.md lists
"Changing the conformance fixture contract" among the changes needing one —
and those cases land against that ADR rather than being approximated in the
meantime.

An adapter encountering a phase it does not implement SHOULD skip the case and
report it as skipped. It MUST NOT report it as passed.
Expand All @@ -119,3 +138,11 @@ report it as skipped. It MUST NOT report it as passed.

A case that does not cite a `clause` will be questioned in review. Fixtures
exist to pin down prose, not to freeze current implementation behaviour.

`task check:conformance` checks two separate things, and only the first of them
needs an implemented phase. Whether or not a case can be *executed* here, its
metadata is validated: the `id` leads with its phase, `cases.json` and
`metadata.json` agree on that phase, the `clause` resolves to an anchor that
exists in the cited `spec.md`, and every declared `code` appears in a
diagnostics table reachable from the family's own — at the phase that table
assigns it. A `semantic` fixture is skipped for execution but not for this.
15 changes: 15 additions & 0 deletions conformance/blueprint/v1/cases.json
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,21 @@
"id": "structural-011-traversal-inside-local-reference",
"phase": "structural",
"path": "structural/011-traversal-inside-local-reference"
},
{
"id": "structural-012-invalid-node-name",
"phase": "structural",
"path": "structural/012-invalid-node-name"
},
{
"id": "semantic-001-connection-names-unknown-role",
"phase": "semantic",
"path": "semantic/001-connection-names-unknown-role"
},
{
"id": "semantic-002-connection-cycle",
"phase": "semantic",
"path": "semantic/002-connection-cycle"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# `fromRole: database` names no node — the only other node is `db`. A
# connection cannot reach outside the graph it is written in, so there is
# nowhere else this could have meant. The whole graph is in this one document,
# which is why the check needs no sibling file.
specVersion: v1
kind: BLUEPRINT
metadata:
slug: web-and-database
version: 1
spec:
components:
db:
component: ./components/postgres.yaml
size: general.standard.small
connections: {}
web:
component: ./components/web.yaml
size: general.standard.small
connections:
DATABASE_URL:
fromRole: database
fromOutput: connectionString
parameters: {}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[
{
"code": "ERR_UNKNOWN_ROLE",
"path": "/spec/components/web/connections/DATABASE_URL/fromRole"
}
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"id": "semantic-001-connection-names-unknown-role",
"phase": "semantic",
"expected": "fail",
"clause": "specifications/blueprint/v1/spec.md#connections",
"summary": "A connection whose fromRole names no node in this blueprint is rejected."
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Mutual discovery: `api` reads `web`'s address and `web` reads `api`'s. Every
# output here is a function of its own node, so a two-pass resolver would
# settle this without complaint — the graph is rejected because a
# specification that permits cycles obliges every implementation to be that
# resolver, and forecloses any later rule needing an order.
#
# The walk is reported from `api`, the lexicographically smallest node in the
# cycle, so that two implementations finding this cycle name it identically:
# api -> web -> api.
specVersion: v1
kind: BLUEPRINT
metadata:
slug: mutual-discovery
version: 1
spec:
components:
api:
component: ./components/api.yaml
size: general.standard.small
connections:
WEB_URL:
fromRole: web
fromOutput: publicUrl
web:
component: ./components/web.yaml
size: general.standard.small
connections:
API_URL:
fromRole: api
fromOutput: privateAddress
parameters: {}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[
{
"code": "ERR_DEPENDENCY_CYCLE",
"path": "/spec/components/api/connections"
}
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"id": "semantic-002-connection-cycle",
"phase": "semantic",
"expected": "fail",
"clause": "specifications/blueprint/v1/spec.md#connections",
"summary": "A cyclic connection graph is rejected, and the walk is reported from its smallest node."
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# `Web_Server` is not a node name. The grammar is the one `metadata.slug`
# uses, so a name that reads as an identifier in one language and not in
# another cannot be what a connection's `fromRole` refers to.
specVersion: v1
kind: BLUEPRINT
metadata:
slug: web-and-database
version: 1
spec:
components:
Web_Server:
component: ./components/web.yaml
size: general.standard.small
connections: {}
parameters: {}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[
{
"code": "ERR_INVALID_VALUE",
"path": "/spec/components"
}
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"id": "structural-012-invalid-node-name",
"phase": "structural",
"expected": "fail",
"clause": "specifications/blueprint/v1/spec.md#components",
"summary": "A node name outside the slug grammar is rejected."
}
95 changes: 95 additions & 0 deletions conformance/component/v1/cases.json
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,101 @@
"id": "structural-005-nested-unknown-field",
"phase": "structural",
"path": "structural/005-nested-unknown-field"
},
{
"id": "structural-006-non-service-with-endpoints",
"phase": "structural",
"path": "structural/006-non-service-with-endpoints"
},
{
"id": "structural-007-non-service-with-empty-endpoints",
"phase": "structural",
"path": "structural/007-non-service-with-empty-endpoints"
},
{
"id": "structural-008-job-without-command",
"phase": "structural",
"path": "structural/008-job-without-command"
},
{
"id": "structural-009-cron-workload",
"phase": "structural",
"path": "structural/009-cron-workload"
},
{
"id": "structural-010-cron-without-schedule",
"phase": "structural",
"path": "structural/010-cron-without-schedule"
},
{
"id": "structural-011-schedule-on-service",
"phase": "structural",
"path": "structural/011-schedule-on-service"
},
{
"id": "structural-012-unpinned-image-reference",
"phase": "structural",
"path": "structural/012-unpinned-image-reference"
},
{
"id": "structural-013-digest-pinned-image-reference",
"phase": "structural",
"path": "structural/013-digest-pinned-image-reference"
},
{
"id": "structural-014-git-source-without-build",
"phase": "structural",
"path": "structural/014-git-source-without-build"
},
{
"id": "structural-015-public-service-without-readiness",
"phase": "structural",
"path": "structural/015-public-service-without-readiness"
},
{
"id": "structural-016-private-service-without-readiness",
"phase": "structural",
"path": "structural/016-private-service-without-readiness"
},
{
"id": "structural-017-relative-mount-path",
"phase": "structural",
"path": "structural/017-relative-mount-path"
},
{
"id": "structural-018-generated-input-not-sensitive",
"phase": "structural",
"path": "structural/018-generated-input-not-sensitive"
},
{
"id": "structural-019-connection-input-with-ui",
"phase": "structural",
"path": "structural/019-connection-input-with-ui"
},
{
"id": "structural-020-user-input-without-ui",
"phase": "structural",
"path": "structural/020-user-input-without-ui"
},
{
"id": "structural-021-derived-output-with-value",
"phase": "structural",
"path": "structural/021-derived-output-with-value"
},
{
"id": "structural-022-contract-both-output-forms",
"phase": "structural",
"path": "structural/022-contract-both-output-forms"
},
{
"id": "semantic-001-floating-tag-image-reference",
"phase": "semantic",
"path": "semantic/001-floating-tag-image-reference"
},
{
"id": "semantic-002-probe-names-unknown-endpoint",
"phase": "semantic",
"path": "semantic/002-probe-names-unknown-endpoint"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# The tag satisfies the structural grammar — it is a tag, not a bare name —
# but it floats. The blocklist is held in the semantic phase rather than in a
# pattern so it can be extended without a major version.
specVersion: v1
kind: COMPONENT
metadata:
version: 1
spec:
workload:
kind: SERVICE
source:
type: IMAGE
ref: ghcr.io/musher-dev/api:nightly
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[
{
"code": "ERR_UNPINNED_IMAGE",
"path": "/spec/workload/source/ref"
}
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"id": "semantic-001-floating-tag-image-reference",
"phase": "semantic",
"expected": "fail",
"clause": "specifications/component/v1/spec.md#source",
"summary": "An image reference whose tag is in the floating set is rejected."
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# The probe targets "api", but the only endpoint is "web". JSON Schema cannot
# catch this: the endpoint names are mapping keys elsewhere in the document,
# and no keyword constrains a value against a sibling's keys.
specVersion: v1
kind: COMPONENT
metadata:
version: 1
spec:
workload:
kind: SERVICE
source:
type: IMAGE
ref: nginx:1.29.4-alpine
endpoints:
web:
containerPort: 8080
protocol: HTTP
visibility: PUBLIC
health:
readiness:
path: /healthz
endpoint: api
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[
{
"code": "ERR_UNKNOWN_ENDPOINT",
"path": "/spec/workload/health/readiness/endpoint"
}
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"id": "semantic-002-probe-names-unknown-endpoint",
"phase": "semantic",
"expected": "fail",
"clause": "specifications/component/v1/spec.md#health",
"summary": "A probe naming an endpoint the workload does not declare is rejected."
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Only a SERVICE may expose an endpoint. A WORKER that declares one has
# either the wrong kind or a misplaced endpoint, and both are worth saying out
# loud — accepting it silently leaves an author expecting traffic that will
# never arrive.
specVersion: v1
kind: COMPONENT
metadata:
version: 1
spec:
workload:
kind: WORKER
command: /bin/consume
source:
type: IMAGE
ref: ghcr.io/musher-dev/worker:1.4.0
endpoints:
web:
containerPort: 8080
protocol: HTTP
visibility: PRIVATE
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[
{
"code": "ERR_INVALID_VALUE",
"path": "/spec/workload/endpoints"
}
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"id": "structural-006-non-service-with-endpoints",
"phase": "structural",
"expected": "fail",
"clause": "specifications/component/v1/spec.md#workload",
"summary": "A WORKER declaring an endpoint is rejected rather than having it ignored."
}
Loading