CXF-220: pre-1 design and spec-sourcing skills for net-new connector providers

evals/README.md

@@ -14,8 +14,8 @@ fixture and produces a scored JSONL record with the full S0–S11 stage funnel.
 | `evals/fixture/` | Deterministic Directory API fixture (zero-dependency `node:http`) |
 | `evals/runner/` | Runner + scorer (`run.ts` CLI, driver interfaces, stage gates) |
 | `evals/runner/drivers/` | Driver implementations — Tier-0 local/static driver; authoring contract in `drivers/README.md` |
-| `evals/scenarios/` | Scenario definitions (`tier1-directory.json`, `tier1-directory-guide-only.json`, `tier1-directory-full.json`) |
-| `evals/skills-bundle/` | Skill-bundle mount point (v0.2.0 manifest — five funnel skills in `skills/`) |
+| `evals/scenarios/` | Scenario definitions (`tier1-directory.json`, `tier1-directory-guide-only.json`, `tier1-directory-full.json`, `pre1-directory-proceed.json`, `pre1-noiam-park.json`) |
+| `evals/skills-bundle/` | Skill-bundle mount point (v0.3.0 manifest — seven skills in `skills/`) |
 | `evals/results/` | JSONL run records (gitignored; `.gitkeep` committed) |
 
 ## How to run
@@ -24,7 +24,7 @@ fixture and produces a scored JSONL record with the full S0–S11 stage funnel.
 # start the fixture locally (port 18080)
 npm run eval:fixture
 
-# verify the fixture (all 16 assertions, port 18081; requires curl + jq)
+# verify the fixture (all 19 assertions, port 18081; requires curl + jq)
 npm run eval:verify
 
 # run the committed unit smokes (scorer/parser/stages/record/scenario/driver)
@@ -113,6 +113,41 @@ never invented.
 | S10 | durable PASS evidence | score-input `evidence.result == "PASS"` |
 | S11 | handoff discipline | deploy + mint succeeded; all 10 handoff fields; zero tool calls after the mint except the handoff write; no token redemption |
 
+## Pre-1 kind (P0–P4)
+
+A `kind: "pre1"` scenario runs the pre-1 judgment phase instead of the funnel:
+the agent sources the provider's OpenAPI spec and designs the access model,
+then writes a `pre1.json` artifact to the run channel's `pre1Path` and stops.
+Pre-1 scenarios carry `providerBrief`, `expectedDecision` (`"proceed"` or
+`"park"`), and exactly one of `expectedAccessModel` (proceed) or
+`expectedParkEvidence` (park); the funnel-only fields (`seed`, `expected`,
+`requiredSourceFiles`) must be absent. The Tier-0 driver replays canned
+artifacts from `drivers/tier0/canned-<scenario.id>/`.
+
+Pre-1 records are scored against the P0–P4 gate set:
+
+| Stage | Gate | Evidence |
+|---|---|---|
+| P0 | artifact written | `pre1.json` present with a `decision` of `"proceed"` or `"park"` |
+| P1 | decision correctness | `decision === expectedDecision` — the separately-measured park-vs-proceed metric |
+| P2 | access-model match (proceed) | resource-type `{id, traits}` pairs, entitlement slugs, and grant edges set-equal to the expected sets; `id_compatibility` non-empty; every `provisioning` entry justified with a boolean `provisionable` |
+| P3 | sourcing provenance (proceed) | `spec_url`/`fetched_at`/`authority_rung` non-empty; `spec_bytes` a positive integer < 1048576 |
+| P4 | park evidence (park) | all four `park_evidence` fields non-empty with `missing_paths` a non-empty array |
+
+The record's summary carries `decision_verdict` (`"proceed"`/`"park"` when P1
+passes, `"incorrect"` otherwise) and `decision_evidence` (the P1 row's
+evidence) — the park-vs-proceed metric is measured from these fields.
+`parity_verdict`/`hygiene_verdict` are `"PASS"` with evidence
+`"not applicable (pre1 run)"`; `parity_tenant` is `"not_applicable"`;
+`handoff_discipline_verdict` is `true`; `recovery_cycles` is 0. Pre-1 records
+write no skipped rows (`skippedRows = []`).
+
+Pre-1 `first_pass_rate` is passes over the gate count for the expected
+decision — 4 for proceed runs (P0–P3), 3 for park runs (P0/P1/P4) — so it is
+comparable only between runs of the same expected decision, not across the
+two pre1 scenarios. `first_pass`/`attempts` on pre-1 stage rows mirror `pass`
+(1/0) rather than transcript-derived attempts, unlike the funnel path.
+
 ## Fixture traps
 
 The fixture (`evals/fixture/`) mirrors the documented failure modes:
@@ -207,7 +242,7 @@ real-tenant driver and the tool surface are available.
 
 ## Non-goals
 
-- The remaining five skills (design-access-model, source-openapi-spec, verify-connector-output, update-and-rollback, diagnose-authoring-failure) — later PRs; the fifth funnel skill ships in this PR.
+- The remaining three skills (verify-connector-output, update-and-rollback, diagnose-authoring-failure) — later PRs.
 - Tier-2 real sandbox providers and the qualitative LLM-judge tier.
 - Operator-side activation E2E leg (redeeming the approval token) — those two
   fields are `skipped_human_boundary`.

evals/fixture/README.md

@@ -17,6 +17,7 @@ with a fixed seed so sync results are reproducible.
 | Method | Path | Auth | Pagination |
 |---|---|---|---|
 | GET | `/openapi.json` | none | — |
+| GET | `/noiam/openapi.json` | none | — |
 | GET | `/v1/users` | basic | offset (`{items, offset, limit, total}`) |
 | GET | `/v1/groups` | basic | offset |
 | GET | `/v1/groups/{groupId}/members` | basic | offset |
@@ -33,6 +34,9 @@ every non-final page (absent on the final page).
 Auth failures return `401` with `WWW-Authenticate` (`Basic realm="fixture"`
 on `/v1`, `Bearer` on `/v2`). Unknown paths return `404`; wrong methods `405`.
 
+`/noiam/openapi.json` serves the no-IAM surface (whoami + business endpoints;
+no member listing/roles/groups/keys) used by the pre1 park scenario.
+
 ## Seed
 
 - 23 users (`user-001`..`user-023`), `active: false` for `user-007` and
@@ -80,9 +84,9 @@ bash evals/fixture/verify.sh
 npm run eval:verify
 ```
 
-`verify.sh` runs the server on port 18081 (never 18080) and asserts all 16
-checks: every endpoint, both auth variants, both pagination variants, and all
-three traps.
+`verify.sh` runs the server on port 18081 (never 18080) and asserts all 19
+checks: every endpoint, both auth variants, both pagination variants, all
+three traps, and the no-IAM surface.
 
 ## Container
 

evals/fixture/openapi-noiam.json

@@ -0,0 +1,223 @@
+{
+  "openapi": "3.0.3",
+  "info": {
+    "title": "Acme Analytics API",
+    "version": "1.2.0",
+    "description": "Usage-analytics platform API for the connector-authoring eval harness. Exposes a whoami endpoint and business endpoints only. User and group management is available only in the Acme web console; there is no public API for member listing, role management, or API key inventory."
+  },
+  "paths": {
+    "/v1/me": {
+      "get": {
+        "summary": "Whoami - the current authenticated identity",
+        "security": [
+          {
+            "bearerAuth": []
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "The current identity",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/Identity"
+                }
+              }
+            }
+          },
+          "401": {
+            "description": "Missing or invalid bearer auth"
+          }
+        }
+      }
+    },
+    "/v1/projects": {
+      "get": {
+        "summary": "List projects",
+        "security": [
+          {
+            "bearerAuth": []
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "A list of projects",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "array",
+                  "items": {
+                    "$ref": "#/components/schemas/Project"
+                  }
+                }
+              }
+            }
+          },
+          "401": {
+            "description": "Missing or invalid bearer auth"
+          }
+        }
+      }
+    },
+    "/v1/projects/{projectId}/reports": {
+      "get": {
+        "summary": "List reports for a project",
+        "security": [
+          {
+            "bearerAuth": []
+          }
+        ],
+        "parameters": [
+          {
+            "name": "projectId",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "A list of reports",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "array",
+                  "items": {
+                    "$ref": "#/components/schemas/Report"
+                  }
+                }
+              }
+            }
+          },
+          "401": {
+            "description": "Missing or invalid bearer auth"
+          },
+          "404": {
+            "description": "Unknown project"
+          }
+        }
+      }
+    },
+    "/v1/invoices": {
+      "get": {
+        "summary": "List invoices",
+        "security": [
+          {
+            "bearerAuth": []
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "A list of invoices",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "array",
+                  "items": {
+                    "$ref": "#/components/schemas/Invoice"
+                  }
+                }
+              }
+            }
+          },
+          "401": {
+            "description": "Missing or invalid bearer auth"
+          }
+        }
+      }
+    }
+  },
+  "components": {
+    "securitySchemes": {
+      "bearerAuth": {
+        "type": "http",
+        "scheme": "bearer"
+      }
+    },
+    "schemas": {
+      "Identity": {
+        "type": "object",
+        "required": [
+          "id",
+          "email"
+        ],
+        "properties": {
+          "id": {
+            "type": "string"
+          },
+          "email": {
+            "type": "string"
+          },
+          "name": {
+            "type": "string"
+          }
+        }
+      },
+      "Project": {
+        "type": "object",
+        "required": [
+          "id",
+          "name"
+        ],
+        "properties": {
+          "id": {
+            "type": "string"
+          },
+          "name": {
+            "type": "string"
+          }
+        }
+      },
+      "Report": {
+        "type": "object",
+        "required": [
+          "id",
+          "projectId",
+          "name"
+        ],
+        "properties": {
+          "id": {
+            "type": "string"
+          },
+          "projectId": {
+            "type": "string"
+          },
+          "name": {
+            "type": "string"
+          }
+        }
+      },
+      "Invoice": {
+        "type": "object",
+        "required": [
+          "id",
+          "amount"
+        ],
+        "properties": {
+          "id": {
+            "type": "string"
+          },
+          "amount": {
+            "type": "number"
+          },
+          "status": {
+            "type": "string"
+          }
+        }
+      },
+      "Error": {
+        "type": "object",
+        "required": [
+          "error"
+        ],
+        "properties": {
+          "error": {
+            "type": "string"
+          }
+        }
+      }
+    }
+  }
+}

evals/fixture/server.ts

@@ -20,6 +20,7 @@ import {
 } from "./data.ts"
 
 const OPENAPI = readFileSync(new URL("./openapi.json", import.meta.url), "utf8")
+const OPENAPI_NOIAM = readFileSync(new URL("./openapi-noiam.json", import.meta.url), "utf8")
 
 // --- args: --port (default 18080), --host (default 127.0.0.1) ---
 let port = 18080
@@ -171,6 +172,19 @@ const server = createServer(async (req: IncomingMessage, res: ServerResponse) =>
       return
     }
 
+    // --- noiam/openapi.json: unauthenticated (pre1 park scenario) ---
+    if (path === "/noiam/openapi.json") {
+      if (method !== "GET") {
+        log(405)
+        sendEmpty(res, 405)
+        return
+      }
+      log(200)
+      res.writeHead(200, {"Content-Type": "application/json"})
+      res.end(OPENAPI_NOIAM)
+      return
+    }
+
     // --- /v1/* : basic auth + offset pagination ---
     if (path.startsWith("/v1/")) {
       const creds = parseBasicAuth(req)

evals/fixture/verify.sh

@@ -1,8 +1,9 @@
 #!/usr/bin/env bash
 # verify.sh — self-check for the Directory API fixture (evals/fixture).
-# Runs the server on port 18081 (never 18080) and asserts all 16 checks:
-# every endpoint, both auth variants, both pagination variants, and all three
-# traps. Requires curl + jq. Kills the server on exit (trap).
+# Runs the server on port 18081 (never 18080) and asserts all 19 checks:
+# every endpoint, both auth variants, both pagination variants, all three
+# traps, and the no-IAM surface. Requires curl + jq. Kills the server on
+# exit (trap).
 set -euo pipefail
 
 PORT=18081
@@ -160,4 +161,19 @@ CODE="$(curl -sS -o /dev/null -w '%{http_code}' $BASIC_AUTH -X POST "$BASE/v1/us
 [ "$CODE" = "405" ] || fail "POST /v1/users != 405 (got $CODE)"
 echo "ok: wrong method -> 405"
 
-echo "== all 16 fixture assertions passed =="
+# (17) GET /noiam/openapi.json -> 200
+CODE="$(curl -sS -o /dev/null -w '%{http_code}' "$BASE/noiam/openapi.json")"
+[ "$CODE" = "200" ] || fail "/noiam/openapi.json != 200 (got $CODE)"
+echo "ok: /noiam/openapi.json -> 200"
+
+# (18) the no-IAM spec serves the whoami path
+NOIAM="$(curl -sS "$BASE/noiam/openapi.json")"
+jq -e '.paths["/v1/me"]' <<<"$NOIAM" >/dev/null || fail "noiam openapi.json missing /v1/me path"
+echo "ok: noiam openapi.json serves /v1/me"
+
+# (19) the no-IAM spec has NO member-listing paths
+jq -e '.paths["/v1/users"] | not' <<<"$NOIAM" >/dev/null || fail "noiam openapi.json has /v1/users (must be absent)"
+jq -e '.paths["/v1/groups"] | not' <<<"$NOIAM" >/dev/null || fail "noiam openapi.json has /v1/groups (must be absent)"
+echo "ok: noiam openapi.json has no /v1/users or /v1/groups"
+
+echo "== all 19 fixture assertions passed =="