CXF-220: pre-1 design and spec-sourcing skills for net-new connector providers

[btipling]

Summary

Net-new connector providers need two judgment steps before any source is written: decide whether the provider's API actually has an IAM surface worth connecting, and design the access model (resource types, traits, entitlements, grants) from the sourced spec. This PR ships those two steps as agent skills — design-access-model and source-openapi-spec — and makes the eval harness measure the park-vs-proceed decision separately from the rest of the funnel, so a provider that should be parked (no member-listing API) is scored on the quality of its park evidence, not on a connector that was never built.

Linear

CXF-220: CXF-70 PR 5: design + spec sourcing skills (3, 4)

Linear requirements

This PR meets all requirements of CXF-220.

Requirement	Status	Evidence
design-access-model skill: API → resource types/traits/entitlements/grants, stable-ID rules, sync-only-vs-provisionable with mandatory "because the API lacks X" justification, ID-compatibility output contract, port minus repo assumptions, WithExternalID DEPRECATED	✅ Met	skills/design-access-model/SKILL.md L1-90 (decision tables L21-44, stable-ID rules L46-49, output contract L56-64, anti-patterns L79-84)
source-openapi-spec skill: 5-rung authority ladder, IAM go/no-go gate, park-with-evidence success outcome, in-app spec-weight twist (262144/1048576 caps)	✅ Met	skills/source-openapi-spec/SKILL.md L1-81 (ladder L14-32, go/no-go gate L34-47, spec-weight L49-54, output contract L56-61)
Pre-computed decision tables (trait selection, sync-only-vs-provisionable, base-url required-vs-literal), every row condition → decision → mandatory justification sentence	✅ Met	skills/design-access-model/SKILL.md L21-44 (tables a/b/c)
Park-vs-proceed correctness measured separately in the eval	✅ Met	evals/runner/stages.ts L358+ (P1 gate decision === expectedDecision), evals/runner/score.ts L167-200 (decision_verdict/decision_evidence), evals/runner/record.ts L37
Done-criterion: fixture net-new provider reaches correct access model + sourced spec; park decisions correct with evidence	✅ Met	evals/scenarios/pre1-directory-proceed.json, evals/scenarios/pre1-noiam-park.json, evals/fixture/openapi-noiam.json, Tier-0 canned replays in evals/runner/drivers/tier0/canned-pre1-*/

Release Notes

No user-facing change.

What changed

skills/design-access-model/SKILL.md (new)

Where: whole file, L1-90.

What changed

• New pre-1 skill: maps a provider API to resource types/traits/entitlements/grants, with three pre-computed decision tables (trait selection, sync-only-vs-provisionable, base-url config-vs-literal), stable-ID rules, and the access_model half of the pre1.json output contract.
• Port of baton-admin design-baton-access-model minus repo assumptions; handoffs repoint to in-app skills (source-openapi-spec, write-connector-source).

Why

• The Linear requirement: net-new providers need a judgment skill for access-model design before source authoring, with the decision tables pre-computed so the judgment calls are deterministic.

skills/source-openapi-spec/SKILL.md (new)

Where: whole file, L1-81.

What changed

• New pre-1 skill: the 5-rung authority ladder for sourcing an OpenAPI spec, a programmatic IAM go/no-go gate (member/user listing at minimum, verified from the spec's paths), park-with-evidence as a success outcome, and the in-app twist: wc -c the spec and record bytes against the 262144-byte source / 1048576-byte bundle caps.
• Port of the marketplace source-openapi-spec skill; emits the sourcing and park_evidence halves of the pre1.json artifact.

Why

• The Linear requirement: spec sourcing must be authority-ranked and the IAM surface verified programmatically, with parking a provider that lacks an IAM API treated as a correct outcome.

skills/design-access-model/SOURCES.md, skills/source-openapi-spec/SOURCES.md (new)

Where: whole files.

What changed

• Provenance tables pinning the port sources (baton-admin 6fe6886f…, claude-marketplace 0cc5ac2a…, c1 16e0e0fb…, in-repo SDK at 7dc673ac…).

Why

• Repo convention (decision 13): nothing written from model memory; every quoted contract fact verified against a pinned source.

skills/author-in-app-connector/SKILL.md

Where: routing table L93-94, new Pre-S1 checklist L100-103, frontmatter version L4.

What changed

• Routing rows split: pre-1 design (net-new provider) → design-access-model, pre-1 spec sourcing → source-openapi-spec; added a pre-S1 checklist note; version bumped 0.2.0 → 0.2.1.

Why

• The orchestrator must route net-new-provider work to the two new pre-1 skills before the funnel starts.

skills/README.md

Where: table L14-15, header L3.

What changed

• Two new provenance rows for the pre-1 skills; "five funnel skills" → "seven skills".

Why

• Keep the skills index in sync with the shipped set.

evals/fixture/openapi-noiam.json (new)

Where: whole file, 223 lines.

What changed

• A plausible provider spec with only a whoami endpoint and business endpoints (projects, reports, invoices) — no member listing, no roles, no groups, no key inventory.

Why

• The park scenario needs a spec whose IAM surface is genuinely absent, so the agent's park decision is exercised against a real go/no-go gate.

evals/fixture/server.ts

Where: L23 (load), L175-186 (route).

What changed

• Serves the no-IAM spec at GET /noiam/openapi.json, unauthenticated, same pattern as /openapi.json.

Why

• The pre1 park scenario fetches the spec from the fixture; the route must exist for the Tier-0 replay.

evals/fixture/verify.sh

Where: L164-178 (assertions 17-19).

What changed

• Three new assertions: /noiam/openapi.json returns 200, serves /v1/me, and has no /v1/users or /v1/groups paths. 16 → 19 checks.

Why

• The fixture's self-check must prove the no-IAM surface is served and genuinely lacks member-listing paths.

evals/runner/scenario.ts

Where: Scenario interface L31-70, kind dispatch L140-145, funnel branch L145-228, pre1 branch L230-355.

What changed

• New optional kind ("funnel" default | "pre1"); pre1 scenarios carry providerBrief, expectedDecision, and exactly one of expectedAccessModel/expectedParkEvidence; funnel-only fields (seed, expected, requiredSourceFiles) are rejected for pre1; both expected halves are shape-validated at load time.

Why

• The runner needs a second scenario kind for the pre-1 judgment phase; validation must fail at load, not at score time.

evals/runner/stages.ts

Where: StageCtx pre1 fields L40-45, Pre1Artifact types L48-70, PRE1_STAGES L358-527.

What changed

• New P0-P4 gate set: P0 artifact written, P1 decision correctness (the separately-measured park-vs-proceed metric), P2 access-model match (set-equal resource types/traits, entitlement slugs, grant edges; non-empty id_compatibility; every provisioning entry justified with a boolean provisionable), P3 sourcing provenance (spec_bytes positive integer < 1048576), P4 park evidence (all four fields non-empty, missing_paths non-empty array). Every check type-checks the untrusted artifact defensively and never throws.

Why

• The pre-1 phase needs its own deterministic gate set; a malformed artifact must fail its gate, not crash the scorer.

evals/runner/score.ts

Where: ScoreResult L25-27, pre1 branch L167-200.

What changed

• scoreRun branches on kind === "pre1": scores PRE1_STAGES, sets decision_verdict to the expected decision when P1 passes and "incorrect" otherwise, and reports parity/hygiene as "not applicable (pre1 run)".

Why

• The park-vs-proceed metric is the P1 row; the summary must carry it as a first-class field.

evals/runner/run.ts

Where: readBoundedJson L118-160, readPre1Artifact L178-184, channel pre1Path L201, pre1 run path L376-425.

What changed

• Shared bounded read for agent-written artifacts (64 MiB cap, 3-attempt retry, null on missing/oversized/malformed/non-object); readPre1Artifact for the pre1 artifact; the pre1 run path reads the artifact, skips the collector leg, scores P0-P4, and writes the record with decision_verdict.

Why

• The pre1 artifact is agent-written and untrusted; the run must treat a bad artifact as a failed P0, never a crash.

evals/runner/agent.ts

Where: pre1 prompt branch L18-38.

What changed

• buildPrompt returns a pre1 prompt: run the two pre-1 skills in order, fetch the spec, write pre1.json to the channel's pre1Path with the output contract, then driver.complete_run.

Why

• The agent needs the pre1 artifact target, the skill names, and the completion verb in its prompt.

evals/runner/baseline.ts

Where: L77-88.

What changed

• A record whose stage rows do not begin with the S0 funnel row is skipped with a one-line stderr warning, never validated, never fatal.

Why

• Pre-1 records can land in evals/results/ locally without breaking baseline generation; funnel-shaped records still validate exactly as before.

evals/runner/driver.ts

Where: RunChannel.pre1Path L63, AgentRunRequest.scenarioId L75-76.

What changed

• Type-only: the channel carries a pre1Path; the agent request carries an optional scenarioId for canned-set selection.

Why

• The pre1 artifact needs a channel path; the Tier-0 driver needs the scenario id to pick its canned replay set.

evals/runner/drivers/tier0/driver.ts

Where: waitForSpec L89-104, provisioner meta L125-151, canned-dir selection L170-196.

What changed

• Pre-1 readiness polls the spec URL instead of the seeded user count; runAgent selects canned-<scenarioId>/ when it exists (pre1 replays) and falls back to the fixed funnel set; writes pre1.json/score-input.json from the selected set.

Why

• Tier-0 replays of pre1 scenarios need their own canned artifacts; the funnel path stays byte-identical.

evals/runner/record.ts

Where: SummaryLine L36-38.

What changed

• Optional decision_verdict/decision_evidence summary fields.

Why

• The record's summary carries the separately-measured park-vs-proceed metric.

evals/scenarios/pre1-directory-proceed.json, evals/scenarios/pre1-noiam-park.json (new)

Where: whole files.

What changed

• Two pre1 scenarios: a directory provider with a full IAM surface (expected decision proceed with the expected access model) and Acme Analytics with no IAM API (expected decision park with the expected park evidence). Both pin skillBundle {mode: "full", version: "0.3.0"} and the tier1 model/effort pins.

Why

• The Done-criterion needs both decision arms exercised as deterministic scenarios.

evals/runner/drivers/tier0/canned-pre1-directory-proceed/, evals/runner/drivers/tier0/canned-pre1-noiam-park/ (new)

Where: transcript.json, pre1.json, score-input.json in each.

What changed

• Canned replay artifacts: the transcript shows the agent fetching the spec and writing pre1.json via driver.write_file, then driver.complete_run; the canned artifacts are the correct proceed/park answers.

Why

• Tier-0 replays are deterministic by construction; the canned sets encode the correct outcomes the gates must score as passing.

evals/skills-bundle/bundle.json, evals/skills-bundle/README.md

Where: bundle.json L1-12, README v0.3.0 section L5-16.

What changed

• Bundle manifest: 5 → 7 skills, version 0.2.0 → 0.3.0; README documents the v0.3.0 set.

Why

• The eval bundle must mount the two new skills; the manifest is the single source for the version pin.

evals/scenarios/tier1-directory-full.json

Where: skillBundle.version L30.

What changed

• Bundle pin 0.2.0 → 0.3.0.

Why

• The full-mode scenario must pin the mounted bundle version or the load-time drift check fails.

evals/README.md, evals/fixture/README.md

Where: evals/README.md L14-15, L24-27, L116-145 (Pre-1 kind section), L239; fixture/README.md L20, L37-39, L87-90.

What changed

• Document the pre1 kind, the P0-P4 gate set, the decision_verdict metric, the no-IAM fixture surface, and the 19-assertion verify count; Non-goals no longer list the two shipped skills.

Why

• The eval harness docs must describe the new scenario kind and metric.

Excluded

• evals/runner/agent.test.ts, evals/runner/baseline.test.ts, evals/runner/run.test.ts, evals/runner/scenario.test.ts, evals/runner/score.test.ts, evals/runner/stages.test.ts, evals/runner/skills_bundle.test.ts, evals/runner/drivers/tier0/driver.test.ts — test-harness files covering the new pre1 behavior (scenario validation, P-gates, scoring, baseline skip, Tier-0 replays, prompt contract).

[linear-code]

CXF-220

[github-actions]

General PR Review: CXF-220: pre-1 design and spec-sourcing skills for net-new connector providers

Blocking Issues: 0 | Suggestions: 3 | Threads Resolved: 0
Criteria: Criteria status: none loaded - .claude/skills/ci-review.md was not found at trusted base 7dc673ac8661.
Review mode: incremental since 2823a297
View review run

Review Summary

The new commit addresses both prior findings: P4 now enforces ctx.expected.parkEvidence (exact spec_version_checked match plus every expected missing path present, extras allowed) with five new gate tests, and per-skill versions moved out of the test constant into bundle.json so the frontmatter contract is checked against the manifest. I re-scanned the full PR diff for security and correctness — no injection, secret, auth, or resource-exhaustion issues, and the tier0 canned park artifact, the pre1-noiam-park scenario, and the 1.2.0 no-IAM fixture spec all line up so the tightened gate still passes end to end. Three non-blocking suggestions remain, all about diagnosability and residual version drift rather than behavior.

Security Issues

None found.

Correctness Issues

None found.

Suggestions

• evals/runner/stages.ts:520-528 — P4's evidence() wasn't updated for the new expected-comparison, so a failure caused only by a version mismatch or a missing expected path reports every field as set with no reason.
• evals/runner/stages.ts:516 / evals/runner/scenario.ts:242-246 — the loader never ties expectedParkEvidence/expectedAccessModel to expectedDecision, so a park scenario carrying an access model now fails P4 unconditionally at score time instead of erroring at load time.
• evals/skills-bundle/bundle.json:4 — run records still stamp only the top-level bundle version, and nothing forces it to bump when a per-skill entry changes; the README also doesn't document the new field and still says the funnel skills are unchanged.

Prompt for AI agents

Verify each finding against the current code and only fix it if needed.

## Suggestions

In `evals/runner/stages.ts`:
- Around line 520-528: P4's `evidence()` still reports only presence/counts
  (`spec_version_checked=set, missing_paths=N, vendor_doc=set,
  revisit_trigger=set`), so a run that fails the new expected-comparison in
  `check()` produces a fail row with no visible cause. Extend `evidence()` to
  read `ctx.expected?.parkEvidence` and report the version comparison as
  `spec_version=match|mismatch` (or `no-expected` when absent) plus the count
  or list of expected `missing_paths` not found in the artifact, mirroring how
  P2 reports `match`/`mismatch` per dimension.
- Around line 516: `if (expected === undefined) return false` makes a
  misconfigured scenario fail silently at score time. In
  `evals/runner/scenario.ts` around lines 242-246, `loadScenario` only requires
  that exactly one of `expectedAccessModel`/`expectedParkEvidence` be present
  and never checks it against `expectedDecision`. Add a load-time check that
  `expectedDecision: "park"` requires `expectedParkEvidence` and
  `expectedDecision: "proceed"` requires `expectedAccessModel`, throwing a
  descriptive error otherwise, so the mismatch is caught when the scenario is
  loaded instead of surfacing as an unexplained P4 failure.

In `evals/skills-bundle/bundle.json`:
- Around line 4: per-skill versions are now in the manifest and checked against
  SKILL.md frontmatter, but the run record still stamps only the top-level
  `version` (via `buildRunMeta` reading `scenario.skillBundle.version`), and
  nothing requires that top-level version to change when a per-skill entry
  changes. Bumping a skill body plus its frontmatter and its bundle entry keeps
  every test green while records stay stamped `0.3.0`, so runs with different
  skill bodies remain indistinguishable. Either record the per-skill versions in
  the record meta, or add an assertion that the top-level bundle version changes
  whenever any skill entry version changes.

In `evals/skills-bundle/README.md`:
- Around line 15 and line 46: the mount-contract section does not mention the
  new per-skill `version` field in `bundle.json` and still says the scenario
  pins `skillBundle.version` (`0.2.0`), while the v0.3.0 section says "The five
  funnel skills are unchanged" even though `author-in-app-connector` moved to
  `0.2.1` with routing-table and Pre-S1-checklist edits. Update both to describe
  the per-skill `version` field, the current `0.3.0` pin, and the
  author-in-app-connector change.

[github-actions]

No blocking issues found.

[github-actions]

No blocking issues found.

[github-actions]

No blocking issues found.

[btipling]

Round-3 suggestions addressed in 2823a29: (1) baseline.ts skip predicate narrowed to the pre-1 shape (P-led stages) so corrupt funnel records missing S0 still fail the canonical-row check; (2) the dead pre-1 score-input.json write and the two canned-pre1-*/score-input.json files deleted (run.ts builds its own ScoreInput literal on the pre-1 path); (3) skillBundle validation + bundle-version drift check extracted into a single parseSkillBundle(data) helper called by both branches; (4) drivers/README.md updated — RunChannel bullet lists pre1Path/pre1Instructions (with the two-halves contract), AgentDriver bullet documents scenarioId/scenarioKind, and the run-channel obligations add the pre1 rule (write to channel.pre1Path, skip the collector leg). Verified: typecheck + 134/134 tests green, fixture verify 19/19, all three Tier-0 replays green.

[github-actions]

No blocking issues found.

[github-actions]

🟡 Suggestion: evidence() below was not updated for the new expected-comparison, so a run that fails P4 only on spec_version_checked mismatch or a missing expected path still reports spec_version_checked=set, missing_paths=3, vendor_doc=set, revisit_trigger=set — a fail row with no visible reason. P2 already reports match/mismatch per dimension; adding spec_version=match|mismatch and the count of expected paths not found would make P4 failures diagnosable from the record alone. (confidence: high, non-blocking)

[github-actions]

🟡 Suggestion: expected === undefined now hard-fails P4, but loadScenario only enforces that exactly one of expectedAccessModel/expectedParkEvidence is present (scenario.ts:242-246) — it never ties that choice to expectedDecision. A scenario with expectedDecision: "park" plus expectedAccessModel still loads, then scores P4 as an unconditional fail at run time with evidence showing every field set. Rejecting the mismatched pairing in the loader (park ⇒ expectedParkEvidence, proceed ⇒ expectedAccessModel) would turn that into a load-time error. (confidence: medium, non-blocking)

[github-actions]

🟡 Suggestion: moving per-skill versions into the manifest closes the "versions live only in a test constant" gap, but the run record still stamps only the top-level version (buildRunMeta → scenario.skillBundle.version), and nothing forces that top-level bump when a per-skill entry changes. Bumping design-access-model to 0.1.1 (body + frontmatter + this entry) keeps every test green and still stamps records 0.3.0, so two runs with different skill bodies remain indistinguishable in the results. Deriving/asserting the bundle version from the skill entries, or recording the per-skill versions in the record meta, would close it. Separately: the new version field isn't described in evals/skills-bundle/README.md, whose mount contract still says the scenario pins 0.2.0, and its "the five funnel skills are unchanged" line no longer holds now that author-in-app-connector is 0.2.1 with a modified body. (confidence: medium, non-blocking)

[github-actions]

No blocking issues found.