CXF-233: publish the Tier-1 E2E control-group baseline.json (6-run matrix)

[btipling]

What

Commits the locked-v1 evals/results/baseline.json — the Tier-1 E2E control-group reference for the connector-authoring eval regression gate (Linear CXF-233). The *.jsonl records behind it are copied into evals/results/ (gitignored, per the existing convention) and evals/README.md's halt-path section now describes the published reference instead of the blocked state.

The matrix behind it

6 scored runs, 2026-09-04/05, on the private squire driver (ductone/squire-evals, omp agent harness, scenario model together/deepseek-ai/DeepSeek-V4-Flash-0731, reasoningEffort: "high" in both scenario files):

run	mode	funnel (passed stages)	first-pass rate	parity	hygiene
evals-tier1-directory-20260904-204206-352	none	S0,S1,S3,S4,S5,S6,S7,S9,S10,S11 (10/12)	0.50	FAIL	PASS
evals-tier1-directory-20260904-221013-308	none	S0,S1,S3,S4,S6,S7,S9,S10 (8/12)	0.42	FAIL	PASS
evals-tier1-directory-20260904-231553-272	none	S0,S1,S3,S4,S6,S7,S9,S10 (8/12)	0.50	PASS	PASS
evals-tier1-directory-guide-only-20260905-000901-207	guide-only	S0,S1,S3,S4,S6,S7,S9,S10 (8/12)	0.58	PASS	PASS
evals-tier1-directory-guide-only-20260905-012128-950	guide-only	S0,S1,S3,S4,S5,S6,S7,S9,S10 (9/12)	0.67	FAIL	PASS
evals-tier1-directory-guide-only-20260905-015055-103	guide-only	S0,S1,S3,S4,S6,S7,S9,S10 (8/12)	0.58	FAIL	PASS

Generated reference: none — runs 3, passes 0, pass_at_3 0, first-pass mean 0.47; guide-only — runs 3, passes 0, pass_at_3 0, first-pass mean 0.61. Failure pareto: S2 (source-upload PUTs), S8 (config api-token), S11 (activation-mint handoff discipline), S5 (build-run state).

Measurement caveats (documented in the README)

• S2 PUT leg: the gate's PUT-200 check only observes bash curl -X PUT results. The omp agent uploads source files through code-mode programs, so the PUT leg reads 0-with-0 even when uploads succeeded (upload_id + create_draft_source_upload legs pass in all runs). The full-funnel pass_rate floor of 0 is therefore partly structural; per-stage and first-pass-rate fields carry the comparable signal.
• S8 api-token reads EMPTY from c1_connector_service_get (secret-masked responses); base-url/account-email record fine.

Stage gates S0/S1/S3/S4/S6/S7/S9/S10 passed in every run of both arms (0 per-stage failures) — the transcript-level gates are healthy under the squire transport after CXF-233's device-route translation.

Driver-side context (private repo, not part of this PR)

• Driver work landed on ductone/squire-evals branch bjorn/CXF-233/collector-transport-override (on top of the CXF-234 PR head 4fa17e3): re-applied the two collector-leg fixes (arena-transport read instructions + completion override, lost with the first attempt's env) and added the operator-approved MCP-route translation (expandMcpRoutes), which makes omp's device-path MCP calls (xd://mcp__c_dev_execute code-mode programs) visible to the scorer's anti-fabrication transcript gates as their literal c1_connector_authoring_* names.
• The dist.conductorone.com outage that halted the first CXF-233 attempt is over: build_bundle succeeded in all 6 runs (S4/S5 evidence RUN_STATE_SUCCEEDED).

Not in this PR

• The CI regression-gate workflow itself (later PR; the README's consumption rule + the committed baseline is the contract).
• No merge — per repo convention, humans merge.

[linear-code]

CXF-233

[github-actions]

🟡 Suggestion: The regression-gate contract above (lines 231–235) still says the gate "fails if the measured pass rate drops below modes.<mode>.pass_rate", but the published reference has pass_rate: 0 for both arms — that rule can never fire, so the committed contract is a no-op gate. This paragraph itself explains why the full-funnel floor is partly structural (S2's PUT leg is unobservable, S8 reads secret-masked); the consumption rule should be updated in the same PR to key on first_pass_rate_mean and/or per_stage.failures, or to state explicitly that pass_rate gating is inert until S2/S8 are instrumented. (confidence: high)

[github-actions]

🟡 Suggestion: scenarios records two ids (tier1-directory, tier1-directory-guide-only), which contradicts the unchanged README description at evals/README.md:200-201 — "scenario tier1-directory × skill-bundle modes {none, guide-only} × 3 runs each". The guide-only arm actually came from a separate scenario file (evals/scenarios/tier1-directory-guide-only.json, mode: "guide-only"), and the generator enforces one scenario per mode group, so the README's single-scenario phrasing (and its (CXF-217) attribution) should be corrected now that the data makes the mismatch visible. (confidence: high)

[github-actions]

🟡 Suggestion: This file becomes a committed contract artifact that a future CI gate will consume, but nothing validates it — evals/runner/baseline.test.ts only exercises the generator against temp dirs, and the source *.jsonl records are gitignored and live in a private repo, so the committed JSON is unreproducible here. Consider a small check that loads evals/results/baseline.json and asserts the locked v1 shape (schema_version === 1, required top-level keys, both matrix modes present, per_stage covering S0–S11) so drift or a hand-edit is caught before the gate PR lands. (confidence: medium)

[github-actions]

General PR Review: CXF-233: publish the Tier-1 E2E control-group baseline.json (6-run matrix)

Blocking Issues: 0 | Suggestions: 3 | Threads Resolved: 0
Criteria: Criteria status: none loaded - .claude/skills/ci-review.md was not found at trusted base 335176ba9a2a.
Review mode: full
View review run

Review Summary

Scanned the full PR diff (evals/README.md, new evals/results/baseline.json) for security and correctness. The committed baseline is internally consistent and matches what evals/runner/baseline.ts would emit: key order, sorted mode/scenario/run-id ordering, generated_at as the newest started_at, per-stage failure counts, the 21-failure pareto shares, and both first_pass_rate_mean values (17/36 and 22/36) all reconcile with the 6-run matrix in the PR description. No security or correctness issues found; the three suggestions concern the documented gate contract, a README/data mismatch, and the absence of any check on the committed artifact.

Security Issues

None found.

Correctness Issues

None found.

Suggestions

• evals/README.md:231-235 — the regression-gate contract keys on modes.<mode>.pass_rate, which is 0 in both arms, so the documented gate can never fail.
• evals/results/baseline.json:6-9 — scenarios lists two ids, contradicting the unchanged README claim of a single tier1-directory scenario crossed with two modes (evals/README.md:200-201).
• evals/results/baseline.json:2 — no test or CI check validates the committed baseline against the locked v1 schema; the generator smokes only cover temp-dir fixtures.

Prompt for AI agents

Verify each finding against the current code and only fix it if needed.

## Suggestions

In `evals/README.md`:
- Around line 231-235: The "Regression-gate contract" paragraph says the CI gate fails if
  the measured pass rate drops below `modes.<mode>.pass_rate` in the committed
  `evals/results/baseline.json`. Both modes in the newly committed baseline have
  `pass_rate` 0 and `pass_at_3` 0, so no rerun can ever drop below the reference and the
  documented gate is inert. The same PR's "Reference status" paragraph explains why the
  full-funnel floor is partly structural: S2's PUT-200 leg only observes bash `curl -X PUT`
  while the agent uploads via code-mode programs, and S8's `api-token` reads EMPTY because
  of secret masking. Update the consumption rule so it keys on the fields that actually
  carry signal — `modes.<mode>.first_pass_rate_mean` and/or the `per_stage` failure counts
  — or state explicitly that `pass_rate` gating stays inert until S2 and S8 are
  instrumented, so the later gate PR does not implement a no-op check.
- Around line 200-201: The Baseline section describes the matrix as six scored Tier-1 runs,
  scenario `tier1-directory` crossed with skill-bundle modes none and guide-only, 3 runs
  each. The committed `baseline.json` `scenarios` array contains two ids,
  `tier1-directory` and `tier1-directory-guide-only`, because the guide-only arm runs from
  its own scenario file (`evals/scenarios/tier1-directory-guide-only.json`, mode
  guide-only), and `evals/runner/baseline.ts` requires exactly one scenario per mode group.
  Reword to name both scenarios and which mode each supplies, and update the stale
  `(CXF-217)` attribution to reflect that the matrix was produced under CXF-233.

In `evals/results/baseline.json`:
- Around line 2: This file is now a committed contract artifact that a future CI regression
  gate will consume, but nothing in the repo validates it. `evals/runner/baseline.test.ts`
  only runs the generator against temporary fixture directories, and the source `*.jsonl`
  records are gitignored (`.gitignore` entry `evals/results/*.jsonl`) and live in a private
  repo, so the committed JSON cannot be regenerated or diffed in CI. Add a small test that
  reads `evals/results/baseline.json` and asserts the locked v1 shape: `schema_version`
  equals 1; `generated_at`, `model`, `reasoning_effort`, `scenarios`, `modes`, and `pareto`
  all present; both none and guide-only mode groups present, each carrying `run_ids`,
  `runs`, `passes`, `pass_rate`, `pass_at_3`, and `first_pass_rate_mean`; and `per_stage`
  covering all of S0 through S11. This catches hand-edits or schema drift before the gate
  PR depends on the file.

[github-actions]

No blocking issues found.