Skip to content

CXF-222: skills README + SOURCES.md provenance delta - #16

Merged
btipling merged 1 commit into
mainfrom
am/src-3naou/src-3naou-att1-impl
Sep 5, 2026
Merged

CXF-222: skills README + SOURCES.md provenance delta#16
btipling merged 1 commit into
mainfrom
am/src-3naou/src-3naou-att1-impl

Conversation

@btipling

@btipling btipling commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Closes https://linear.app/ductone/issue/CXF-222

Final PR of the measurable-skills batch: closes the remaining provenance rows and adds the eval-evidence section to the skills README.

Changes

  • SOURCES.md contract rows (all ten skills): each skills/<name>/SOURCES.md gains two rows — the SDK contract (baton/*.d.ts @ git tag v0.0.26, sync commit 01a69d8d) with the locked runtime_pin_matched wording, and the baton-axiomatic DSL contract (docs/DSL.md + runtime/baton/*.d.ts @ v0.0.26). Existing pinned rows are untouched.
  • SOURCES.md knowledge rows (three skills): design-access-model names the worked builds (baton-axiomatic-openrouter#1, baton-axiomatic-litellm#1, baton-axiomatic-shopify); source-openapi-spec names the three spec-evidence rows (OpenRouter official, LiteLLM runtime-generated, Shopify vendored minimal); write-connector-source names the baton-admin DSL skills (author-js-dsl-connector, author-auth-config-surface).
  • README: intro tightened to the issue's model terms ("authored against the v0.0.26 DSL contract and the 23-tool tenant MCP surface; validated by the eval suite"); new ## Eval evidence section.

Eval evidence carry-forward

No live pass-rate numbers are reported because none exist: the Tier-1 baseline E2E (baseline vs with-skills pass rates on a real tenant) is structurally blocked — the public repo ships only the Tier-0 canned driver, and the c1-side MCP surface on eval environments exposes no c1_connector_authoring_* tools. This follows the carry-forward rule established in the merged PRs #10/#11. The README cites the deterministic, committed, executable evidence (Tier-0 scored replay, npm run eval:test smokes, scorer/fixture harness).

Validation

  • npm run typecheck — exit 0
  • npm run eval:test — 139 tests, 0 fail

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
@linear-code

linear-code Bot commented Sep 5, 2026

Copy link
Copy Markdown

CXF-222

Comment thread skills/README.md
The ten skills shipped in this batch: the five funnel skills and two
pre-1 skills authored against the v0.0.26 DSL contract and the 23-tool
tenant MCP surface, plus three post-funnel and cross-cutting skills
pre-1 skills authored against the v0.0.26 DSL contract and the 23-tool tenant MCP surface; validated by the eval suite, plus three post-funnel and cross-cutting skills

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion: the inserted ; validated by the eval suite, splits the "five funnel skills and two pre-1 skills … plus three post-funnel …" list mid-sentence, so plus three post-funnel … now dangles after a semicolon clause. The line also runs unwrapped while the rest of the paragraph wraps at ~70 chars. Confidence: high.

Suggested change
pre-1 skills authored against the v0.0.26 DSL contract and the 23-tool tenant MCP surface; validated by the eval suite, plus three post-funnel and cross-cutting skills
pre-1 skills authored against the v0.0.26 DSL contract and the 23-tool
tenant MCP surface, plus three post-funnel and cross-cutting skills

Consider moving "validated by the eval suite" into its own sentence after the list, so the claim isn't attached only to the funnel/pre-1 subset and reads consistently with the ## Eval evidence section below (which states the Tier-1 baseline is blocked).

Comment thread skills/README.md
The measurable evidence for these skills is the deterministic, committed,
executable eval harness in `evals/`:

- **Tier-0 scored replay** — `npm run eval:run -- --scenario evals/scenarios/tier1-directory.json --driver tier0` replays a committed scenario end-to-end against the local fixture and produces a scored JSONL record of the full S0–S11 funnel (no credentials, no network beyond localhost).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion: the scenario cited as the skills' "measurable evidence" is the no-skills arm — evals/scenarios/tier1-directory.json has "skillBundle": {"mode": "none"} (evals/runner/agent.ts:8 emits the no-bundle prompt section for that mode). tier1-directory-full.json is the mode: "full" arm. Additionally, the tier0 agent driver replays a canned transcript (evals/runner/drivers/tier0/driver.ts readCannedFrom("transcript.json")), so the run scores committed artifacts rather than exercising the skill bodies — evals/README.md:31 says this explicitly ("replays committed artifacts") but this bullet does not. Suggest pointing at tier1-directory-full.json and carrying over the "replays committed artifacts" wording. Confidence: high.

Comment on lines +16 to +18
| Worked build — access-model shape | baton-axiomatic-openrouter#1 | The access-model shape (resource types, traits, entitlements, grants) this worked build demonstrates. |
| Worked build — access-model shape | baton-axiomatic-litellm#1 | The access-model shape (resource types, traits, entitlements, grants) this worked build demonstrates. |
| Worked build — access-model shape | baton-axiomatic-shopify | The access-model shape (resource types, traits, entitlements, grants) this worked build demonstrates. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion: these three rows carry no pin in the "Pin / SHA" column — baton-axiomatic-openrouter#1 / baton-axiomatic-litellm#1 are PR references (mutable until merged, and no merge SHA) and baton-axiomatic-shopify names no ref at all. That breaks this file's own header contract ("Authored against the pinned sources below") and skills/README.md's "names the pinned sources with their SHAs". Add merge/commit SHAs. The three rows also share an identical Source label and identical "What this skill quotes" text, so the table no longer distinguishes what each build contributes. Confidence: high.

| baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. |
| OpenRouter official spec (evidence) | OpenRouter official published spec, vendored verbatim | The authoritative spec evidence for a SaaS provider with an official published spec. |
| LiteLLM runtime-generated spec (evidence) | `pip install "litellm[proxy]==1.92.0"` → `app.openapi()` | The runtime-generated spec evidence for a self-hosted OSS provider. |
| Shopify vendored minimal spec (evidence) | `allengrant/shopify_openapi` @ master, discoveredAt 2026-06-04 | The vendored minimal spec evidence for a provider without an official published spec. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion: @ master is a moving ref, not a pin — the vendored minimal spec this row backs can change under the recorded discoveredAt date without the row changing. Pin the commit SHA of allengrant/shopify_openapi that was actually vendored. The OpenRouter row above (line 15) is similarly unpinned ("official published spec, vendored verbatim" with no version/date/SHA). Confidence: high.

| Authoring proto | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | RPC titles `connector_authoring_get_authoring_guide`, `connector_authoring_list_sdk_types_versions`, `connector_authoring_get_sdk_types`, `connector_authoring_list_authored_catalog_entries`, `connector_authoring_list_drafts`, `connector_authoring_create_draft`; `ListSDKTypesVersionsResponse` fields `default_tag` / `runtime_pin_matched`. |
| Lifecycle doc | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | Step 0 (guide read) and the handoff-table discipline (record every returned ID). |
| In-repo SDK declarations | connector-authoring `b7e8a616cbbb1e336b788f807a3810b08ae00bc7` | The `.d.ts` module surface (`@baton/runtime`, `@baton/helpers`, `@baton/types`) the tagged declarations cover. |
| SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion: this row is the same boilerplate applied to all ten skills, so in read-authoring-contract's own SOURCES.md the trailing "(see read-authoring-contract for the runtime check)" points the reader back at this skill. Drop the cross-reference in this one file. Confidence: high.

Also note this row overlaps the existing "In-repo SDK declarations" row (line 13, pinned at b7e8a616…): both describe the .d.ts module surface the skill is authored against, now at two different pins (a commit vs. tag v0.0.26), leaving it ambiguous which governs. Same duplication exists in the other nine files. Consider merging the rows or narrowing each row's scope.

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

General PR Review: CXF-222: skills README + SOURCES.md provenance delta

Blocking Issues: 0 | Suggestions: 5 | Threads Resolved: 0
Criteria: Criteria status: none loaded - .claude/skills/ci-review.md was not found at trusted base bdff5edd6bbd.
Review mode: full
View review run

Review Summary

Documentation-only change: eleven markdown files (the skills README plus ten SOURCES.md provenance tables). The full PR diff was scanned for security and correctness — no executable code, dependency manifests, generated artifacts, or public API surface changed, so there is nothing that can introduce a vulnerability or runtime regression. I verified the newly documented eval commands against the checked-out harness (package.json scripts, evals/runner/run.ts CLI flags, evals/scenarios/tier1-directory.json, evals/runner/drivers/tier0/driver.ts): the commands are valid and runnable, but the cited scenario is the no-skills arm, and several new provenance rows are unpinned — both non-blocking accuracy issues below.

Security Issues

None found.

Correctness Issues

None found.

Suggestions

  • skills/README.md:4 — the inserted ; validated by the eval suite, breaks the sentence's list structure (plus three post-funnel … now dangles) and the line is left unwrapped against the paragraph's ~70-char convention; the claim also sits oddly next to the "structurally blocked" Tier-1 paragraph added below.
  • skills/README.md:32 — the replay cited as the skills' measurable evidence uses evals/scenarios/tier1-directory.json, which sets "skillBundle": {"mode": "none"} (the no-skills arm; tier1-directory-full.json is mode: "full"), and the tier0 driver replays a canned transcript, so the run scores committed artifacts rather than exercising the skill bodies — evals/README.md:31 states this ("replays committed artifacts") but the new bullet does not.
  • skills/design-access-model/SOURCES.md:16-18 — the three worked-build rows carry no pin (…openrouter#1 / …litellm#1 are mutable PR refs; baton-axiomatic-shopify names no ref), contradicting the file header and the README's "names the pinned sources with their SHAs"; they also share an identical label and rationale.
  • skills/source-openapi-spec/SOURCES.md:17 (and :15) — allengrant/shopify_openapi @ master is a moving ref rather than a pin, and the OpenRouter row records no version, date, or SHA.
  • skills/read-authoring-contract/SOURCES.md:14 — the boilerplate SDK-contract row self-references ("see read-authoring-contract for the runtime check") in that skill's own file, and it overlaps the existing "In-repo SDK declarations" row at a different pin (commit b7e8a616… vs tag v0.0.26), leaving which pin governs ambiguous across all ten files.
Prompt for AI agents
Verify each finding against the current code and only fix it if needed.

## Suggestions

In `skills/README.md`:
- Around line 4: The inserted clause "; validated by the eval suite," splits the sentence
  listing the ten skills, so "plus three post-funnel and cross-cutting skills" now dangles
  after a semicolon. Restore the original comma-joined list ("... and the 23-tool tenant MCP
  surface, plus three post-funnel and cross-cutting skills"), re-wrap the paragraph at ~70
  chars to match the rest of the file, and if the eval-validation claim is wanted, put it in
  its own sentence after the list so it is not scoped only to the funnel/pre-1 subset and does
  not conflict with the "structurally blocked" Tier-1 paragraph later in the file.
- Around line 32: The Tier-0 replay bullet cites `evals/scenarios/tier1-directory.json`, which
  has `"skillBundle": {"mode": "none"}` — the baseline arm that runs with no skill bundle
  mounted (see `evals/runner/agent.ts` `skillBundleSection`). Point at
  `evals/scenarios/tier1-directory-full.json` (`mode: "full"`) if the intent is evidence that
  exercises the skills, or state which arm is being cited. Also carry over the wording used in
  `evals/README.md` line 31 that the Tier-0 driver "replays committed artifacts", since
  `evals/runner/drivers/tier0/driver.ts` replays a canned transcript rather than running an
  agent against the skills.

In `skills/design-access-model/SOURCES.md`:
- Around line 16-18: The three "Worked build — access-model shape" rows have no value in the
  "Pin / SHA" column: `baton-axiomatic-openrouter#1` and `baton-axiomatic-litellm#1` are PR
  references (mutable, no merge SHA) and `baton-axiomatic-shopify` names no ref at all. Add the
  merge/commit SHA for each, matching the pin discipline stated in this file's header and in
  `skills/README.md`. Also differentiate the three rows: they currently share an identical
  Source label and identical "What this skill quotes" text, so the table does not say what each
  build contributes.

In `skills/source-openapi-spec/SOURCES.md`:
- Around line 15-17: The Shopify row pins `allengrant/shopify_openapi` to `@ master`, a moving
  ref — replace it with the commit SHA that was actually vendored (keep the discoveredAt date
  as supplementary). The OpenRouter row (line 15) records no version, date, or SHA for the
  "official published spec, vendored verbatim"; add a retrieval date and, if available, a spec
  version or URL so the evidence is reproducible.

In `skills/read-authoring-contract/SOURCES.md`:
- Around line 13-14: The new SDK-contract row is boilerplate shared across all ten skills, so in
  this file it points the reader back at itself ("see `read-authoring-contract` for the runtime
  check"); drop that cross-reference in this one file. Separately, this row and the existing
  "In-repo SDK declarations" row on line 13 both describe the `.d.ts` module surface the skill is
  authored against but cite different pins (commit `b7e8a616...` vs git tag `v0.0.26`), leaving
  it ambiguous which governs. Merge the two rows or narrow each row's scope so exactly one pin is
  authoritative for the `.d.ts` surface. The same duplication was added to the other nine
  SOURCES.md files and should be resolved the same way.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found.

@btipling
btipling merged commit 335176b into main Sep 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant