[EPIC] Establish Agent Skills conformance and evaluation

[dieterbaier]

As a toolkit maintainer or contributor, I want published skills and generated skill distributions to be validated for structural conformance, reliable triggering, correct routing, and expected behaviour, so that the architecture-knowledge-toolkit can be consumed consistently across Agent Skills-compatible platforms without relying on accidental repository context.

Context

The toolkit already provides engine-independent SKILL.md contracts, semantic contracts, adapters, validators, generators, and reproducible builds. The planned machine-readable catalog (#62), versioned distribution (#63), and consumer CLI (#64) improve discovery and consumption.

What is still missing is an explicit quality model for the skills themselves as portable runtime assets. A skill may be structurally valid while still triggering too broadly, failing to trigger for realistic user language, depending on files outside its package, overlapping ambiguously with a neighbouring skill, or behaving incorrectly when input is incomplete or outside its scope.

This epic establishes a deterministic conformance and evaluation layer without making one AI engine authoritative. Canonical human-readable skills and contracts remain the source of truth; evaluations verify their portable projections and observable behaviour.

Goals

• Define a documented conformance profile for toolkit skills and portable Agent Skills packages.
• Validate skill structure, frontmatter, naming, references, dependencies, and package boundaries.
• Evaluate whether realistic user requests select the intended skill or workflow.
• Test boundaries between neighbouring or overlapping skills.
• Add behavioural scenarios for critical skills and workflows.
• Prevent drift between canonical skills, catalog entries, generated distributions, routers, and adapters.
• Keep deterministic structural checks separate from model-dependent behavioural evaluations.
• Produce results that can be used locally, in CI, and by the release process.

Acceptance Criteria

• A documented toolkit skill conformance profile exists.
• Every publishable skill has a directory name matching its frontmatter name and a non-empty triggering-oriented description.
• Skill metadata required for catalog generation and portable distribution is validated.
• All relative references, scripts, assets, and declared dependencies of a portable skill package resolve within the package or through an explicitly declared external dependency.
• Validation fails when a generated skill package has an undeclared dependency on repository-local files outside the package.
• Published skill packages are checked against the selected Agent Skills-compatible structural specification or a documented compatible validator.
• Every skill has at least representative should-trigger and should-not-trigger examples, either directly or through a referenced evaluation set.
• Skills with neighbouring responsibilities have explicit boundary cases that identify which skill should handle the request.
• A routing evaluation suite maps realistic user requests to the expected skill, router decision, or out-of-scope result.
• Critical skills have behavioural scenarios covering at least:
  • a normal in-scope request;
  • incomplete or ambiguous input;
  • a healthy or negative-result case where the skill must not invent findings;
  • an out-of-scope request that delegates or declines correctly.
• Behavioural evaluations define observable assertions rather than depending only on free-form subjective review.
• Structural conformance checks are deterministic and run in the normal build.
• Model-dependent routing and behavioural evaluations can run separately and report the model, runtime, toolkit version, and evaluation data version used.
• Changes to skill names, descriptions, supported tasks, inputs, outputs, dependencies, or routing metadata require the relevant conformance and routing evaluations to pass.
• The build detects drift between canonical SKILL.md files, catalog entries from [EPIC] Publish a machine-readable knowledge and skill catalog #62, generated portable packages from [EPIC] Publish the toolkit as a versioned consumable distribution #63, generated routers, and engine-specific adapters.
• Evaluation fixtures and expected outcomes are version-controlled and covered by automated tests where deterministic execution is possible.
• CI produces a human-readable conformance and evaluation report with actionable failure messages.

Candidate User Stories

• Define the toolkit skill conformance profile.
• Add a deterministic skill package validator.
• Validate frontmatter, directory names, references, scripts, assets, and dependency declarations.
• Add should-trigger and should-not-trigger fixtures for every skill.
• Add boundary evaluations for ADR, risk, quality-scenario, architecture-impact, bootstrap, and traceability workflows.
• Add a task-oriented routing evaluation suite.
• Add behavioural evaluations for critical architecture workflows.
• Generate a conformance report for contributors and releases.
• Fail CI when canonical skills, catalog metadata, generated packages, routers, or adapters drift.
• Provide contributor guidance for adding or changing a skill and its evaluations.

Relationship to Other Epics

• [EPIC] Publish a machine-readable knowledge and skill catalog #62 provides the machine-readable skill and knowledge metadata used by conformance checks and routing evaluations.
• [EPIC] Publish the toolkit as a versioned consumable distribution #63 publishes only skill packages and distributions that pass the required conformance checks.
• [EPIC] Provide a consumer CLI for toolkit adoption and updates #64 exposes conformance, diagnostics, and optional evaluation commands to toolkit consumers.

Out of Scope

• Making one AI model or agent runtime the normative implementation of a skill.
• Replacing deterministic validators with LLM-based judgement.
• Guaranteeing identical wording or reasoning across different models.
• Measuring architecture quality solely through routing accuracy.
• Copying canonical toolkit knowledge into consuming repositories merely to make evaluations pass.