CXF-222: skills README + SOURCES.md provenance delta - #16
Conversation
Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
| The ten skills shipped in this batch: the five funnel skills and two | ||
| pre-1 skills authored against the v0.0.26 DSL contract and the 23-tool | ||
| tenant MCP surface, plus three post-funnel and cross-cutting skills | ||
| pre-1 skills authored against the v0.0.26 DSL contract and the 23-tool tenant MCP surface; validated by the eval suite, plus three post-funnel and cross-cutting skills |
There was a problem hiding this comment.
🟡 Suggestion: the inserted ; validated by the eval suite, splits the "five funnel skills and two pre-1 skills … plus three post-funnel …" list mid-sentence, so plus three post-funnel … now dangles after a semicolon clause. The line also runs unwrapped while the rest of the paragraph wraps at ~70 chars. Confidence: high.
| pre-1 skills authored against the v0.0.26 DSL contract and the 23-tool tenant MCP surface; validated by the eval suite, plus three post-funnel and cross-cutting skills | |
| pre-1 skills authored against the v0.0.26 DSL contract and the 23-tool | |
| tenant MCP surface, plus three post-funnel and cross-cutting skills |
Consider moving "validated by the eval suite" into its own sentence after the list, so the claim isn't attached only to the funnel/pre-1 subset and reads consistently with the ## Eval evidence section below (which states the Tier-1 baseline is blocked).
| The measurable evidence for these skills is the deterministic, committed, | ||
| executable eval harness in `evals/`: | ||
|
|
||
| - **Tier-0 scored replay** — `npm run eval:run -- --scenario evals/scenarios/tier1-directory.json --driver tier0` replays a committed scenario end-to-end against the local fixture and produces a scored JSONL record of the full S0–S11 funnel (no credentials, no network beyond localhost). |
There was a problem hiding this comment.
🟡 Suggestion: the scenario cited as the skills' "measurable evidence" is the no-skills arm — evals/scenarios/tier1-directory.json has "skillBundle": {"mode": "none"} (evals/runner/agent.ts:8 emits the no-bundle prompt section for that mode). tier1-directory-full.json is the mode: "full" arm. Additionally, the tier0 agent driver replays a canned transcript (evals/runner/drivers/tier0/driver.ts readCannedFrom("transcript.json")), so the run scores committed artifacts rather than exercising the skill bodies — evals/README.md:31 says this explicitly ("replays committed artifacts") but this bullet does not. Suggest pointing at tier1-directory-full.json and carrying over the "replays committed artifacts" wording. Confidence: high.
| | Worked build — access-model shape | baton-axiomatic-openrouter#1 | The access-model shape (resource types, traits, entitlements, grants) this worked build demonstrates. | | ||
| | Worked build — access-model shape | baton-axiomatic-litellm#1 | The access-model shape (resource types, traits, entitlements, grants) this worked build demonstrates. | | ||
| | Worked build — access-model shape | baton-axiomatic-shopify | The access-model shape (resource types, traits, entitlements, grants) this worked build demonstrates. | |
There was a problem hiding this comment.
🟡 Suggestion: these three rows carry no pin in the "Pin / SHA" column — baton-axiomatic-openrouter#1 / baton-axiomatic-litellm#1 are PR references (mutable until merged, and no merge SHA) and baton-axiomatic-shopify names no ref at all. That breaks this file's own header contract ("Authored against the pinned sources below") and skills/README.md's "names the pinned sources with their SHAs". Add merge/commit SHAs. The three rows also share an identical Source label and identical "What this skill quotes" text, so the table no longer distinguishes what each build contributes. Confidence: high.
| | baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | | ||
| | OpenRouter official spec (evidence) | OpenRouter official published spec, vendored verbatim | The authoritative spec evidence for a SaaS provider with an official published spec. | | ||
| | LiteLLM runtime-generated spec (evidence) | `pip install "litellm[proxy]==1.92.0"` → `app.openapi()` | The runtime-generated spec evidence for a self-hosted OSS provider. | | ||
| | Shopify vendored minimal spec (evidence) | `allengrant/shopify_openapi` @ master, discoveredAt 2026-06-04 | The vendored minimal spec evidence for a provider without an official published spec. | |
There was a problem hiding this comment.
🟡 Suggestion: @ master is a moving ref, not a pin — the vendored minimal spec this row backs can change under the recorded discoveredAt date without the row changing. Pin the commit SHA of allengrant/shopify_openapi that was actually vendored. The OpenRouter row above (line 15) is similarly unpinned ("official published spec, vendored verbatim" with no version/date/SHA). Confidence: high.
| | Authoring proto | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | RPC titles `connector_authoring_get_authoring_guide`, `connector_authoring_list_sdk_types_versions`, `connector_authoring_get_sdk_types`, `connector_authoring_list_authored_catalog_entries`, `connector_authoring_list_drafts`, `connector_authoring_create_draft`; `ListSDKTypesVersionsResponse` fields `default_tag` / `runtime_pin_matched`. | | ||
| | Lifecycle doc | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | Step 0 (guide read) and the handoff-table discipline (record every returned ID). | | ||
| | In-repo SDK declarations | connector-authoring `b7e8a616cbbb1e336b788f807a3810b08ae00bc7` | The `.d.ts` module surface (`@baton/runtime`, `@baton/helpers`, `@baton/types`) the tagged declarations cover. | | ||
| | SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | |
There was a problem hiding this comment.
🟡 Suggestion: this row is the same boilerplate applied to all ten skills, so in read-authoring-contract's own SOURCES.md the trailing "(see read-authoring-contract for the runtime check)" points the reader back at this skill. Drop the cross-reference in this one file. Confidence: high.
Also note this row overlaps the existing "In-repo SDK declarations" row (line 13, pinned at b7e8a616…): both describe the .d.ts module surface the skill is authored against, now at two different pins (a commit vs. tag v0.0.26), leaving it ambiguous which governs. Same duplication exists in the other nine files. Consider merging the rows or narrowing each row's scope.
General PR Review: CXF-222: skills README + SOURCES.md provenance deltaBlocking Issues: 0 | Suggestions: 5 | Threads Resolved: 0 Review SummaryDocumentation-only change: eleven markdown files (the skills README plus ten Security IssuesNone found. Correctness IssuesNone found. Suggestions
Prompt for AI agents |
Closes https://linear.app/ductone/issue/CXF-222
Final PR of the measurable-skills batch: closes the remaining provenance rows and adds the eval-evidence section to the skills README.
Changes
skills/<name>/SOURCES.mdgains two rows — the SDK contract (baton/*.d.ts@ git tagv0.0.26, sync commit01a69d8d) with the lockedruntime_pin_matchedwording, and the baton-axiomatic DSL contract (docs/DSL.md+runtime/baton/*.d.ts@ v0.0.26). Existing pinned rows are untouched.design-access-modelnames the worked builds (baton-axiomatic-openrouter#1,baton-axiomatic-litellm#1,baton-axiomatic-shopify);source-openapi-specnames the three spec-evidence rows (OpenRouter official, LiteLLM runtime-generated, Shopify vendored minimal);write-connector-sourcenames the baton-admin DSL skills (author-js-dsl-connector,author-auth-config-surface).## Eval evidencesection.Eval evidence carry-forward
No live pass-rate numbers are reported because none exist: the Tier-1 baseline E2E (baseline vs with-skills pass rates on a real tenant) is structurally blocked — the public repo ships only the Tier-0 canned driver, and the c1-side MCP surface on eval environments exposes no
c1_connector_authoring_*tools. This follows the carry-forward rule established in the merged PRs #10/#11. The README cites the deterministic, committed, executable evidence (Tier-0 scored replay,npm run eval:testsmokes, scorer/fixture harness).Validation
npm run typecheck— exit 0npm run eval:test— 139 tests, 0 fail