Baseline gate: no XP spend before the first submission completes

[devin-ai-integration]

Investigation findings

The pre-baseline spend is not one bug in one file — it is the ordering baked into three places, none of which ever says "measure before you have a ladder result", so each one looks reasonable alone:

• docs/getting-started.md — beat 5 uploads, beat 6 was "First evaluation" ("Run a hosted batch sized by the lab's eval-design binding"). Submission appeared only in the closing block, as an option: "If they want to submit, help them submit."
• AGENTS.md — the loop table puts 2 Evaluate and 3 Read ahead of 10 Submit — the gate, and then says "Steps 1–3 and 6–8 are cheap and fast by design — no gates". For a lab with a baseline that is exactly right; for a lab without one, "cheap" is spend against no reference.
• skills/seed-a-policy/SKILL.md — handed a freshly seeded policy line straight into run-eval: "the first eval goes through run-eval and survey".

Two things the fix had to respect. First, non-negotiable #5 — submission "requires the human's explicit go-ahead, every time" — so "baseline first" cannot mean an automatic submit; it means propose first, and spend nothing until the human's baseline lands. Second, submit's preconditions demand "a verdict-rung ab-compare result against the current baseline", which a baseline submission can never satisfy: that precondition was itself part of the loop that forced an eval before the first submission.

Outside this repo, the same ordering is mirrored in Metta-AI/optimizer-agent (skills/seed-a-new-policy/SKILL.md), in Metta-AI/optimizer-skills (skills/seed-a-new-policy/SKILL.md, starter-pack/SKILL.md, starter-pack/cards/seed.md, playbooks/optimize-policy.md), and — authoritatively, since it is what a live session actually follows — in the Devin playbook "Run an Agent Optimization Loop for a Coworld Policy" (!optimize_policy), whose step 3 is "Diagnose the current policy with a smoke + diagnostic XP eval" with submission at step 10. Those are tracked separately (one issue per mirror repo, and replacement wording handed to the playbook's publisher); this PR is the reference implementation.

The change

A ninth non-negotiable plus a real check, so the rule survives a future edit that reorders the prose:

9. No spend before a baseline. Until this policy line has a completed league
   submission with a real ladder result, the only permitted next step is
   proposing that baseline submission — of the policy exactly as it is. No
   hosted eval, no `coworld xp-request`, no A/B, no experiment, no sweep.

• tools/check_baseline_gate.sh <lab> is the enforcement. It reads the lab's WORKING_CONTEXT.md → ## Baseline submission block and exits 0 only when state: completed and a real ladder result is recorded; 1 = no baseline yet (with the permitted next step printed, specialized per state), 2 = cannot tell (no lab, missing block, unrecognized state) — an unknown baseline is not a baseline. Placeholder text like *(rank + score + date)* is voided, so an untouched template never passes.
• Every spending skill calls it first and refuses on non-zero: run-eval (step 0 of its method, before any body is composed), ab-compare, experiment (before the record is even opened — designing an experiment you may not run is not a useful pause point). That is what makes this a gate rather than an ordering: reordering the docs no longer unblocks the spend, because the skills ask the script.
• skills/submit now distinguishes the two submissions it can perform: the baseline submission carries no evidence preconditions (it is the reference everything later is measured against) and records the gate state; a promotion submission keeps the full evidence bar. Consent is unchanged for both.
• skills/seed-a-policy exits into submit, not run-eval.
• docs/getting-started.md gains beat 6, "The baseline submission — before any measurement spend" (old beat 6 → 7), including the decline path: record state: blocked, say optimization is on hold, and explicitly do not fall back to "we'll just run a small diagnostic instead" — that fallback is the failure mode this beat exists to prevent.
• games/_template/WORKING_CONTEXT.md gains the ## Baseline submission field (state / policy / league / submission / ladder result), so the gate has a per-lab input that a resuming agent can read with no chat history (#7).

Evidence

The repo has no test harness, so the check ships with its own: tools/check_baseline_gate.sh --self-test builds a fixture per state and asserts the exit code that decides whether XP can be spent.

$ tools/check_baseline_gate.sh --self-test
ok   completed + ladder result          exit 0
ok   state none                         exit 1
ok   submission still in flight         exit 1
ok   human declined                     exit 1
ok   completed but no ladder result     exit 1
ok   unrecognized state                 exit 2
ok   no Baseline submission block       exit 2
ok   no such lab                        exit 2
ok   ambiguous: several labs, none named exit 2
baseline gate self-test: all 9 cases pass

Against this repo as it ships (no lab installed), the gate blocks, which is the correct default:

$ tools/check_baseline_gate.sh
baseline gate: BLOCKED — no lab installed, so no baseline can exist.
  Next: tools/add_game.sh <mixin-repo-url>, then onboard the lab.
$ echo $?
2

bash -n clean; the script is plain bash + coreutils (macOS and Linux), matching the existing hooks in tools/.

Treat this as a proposal: every commit in this repo's history is authored by James Boggs (james@softmax.com), so the doctrine here is not mine to change unilaterally — in particular the wording of non-negotiable #9 and the decision to have submit distinguish baseline from promotion submissions are the two calls a maintainer should review before merge.

Harness enforcement (added after the first review round), and what it does not cover

The doc + check_baseline_gate.sh half above binds an agent that reads a skill and chooses to call the script. A live
run showed that is not enough: a session composed "go straight to hosted XP" as a recommended option on its own,
without any file telling it to. So both harnesses now gate at the tool boundary, where the agent gets no say.

• harness/claude-code/settings.json and harness/codex/hooks.json gain a PreToolUse hook on Bash
  (tools/baseline_pretooluse_gate.sh) and a SessionStart hook (tools/baseline_session_start.sh).

• The PreToolUse hook parses tool_input.command, matches only concrete spending forms
  (coworld xp-request, ab-compare, eval-sweep/eval_sweep, eval_request.py, the experiment record.py
  entrypoint), calls the gate, and returns the vendor-supported deny response:

  {"hookSpecificOutput": {"hookEventName": "PreToolUse", "permissionDecision": "deny",
                          "permissionDecisionReason": "..."}}

  Gate exit 1 (no baseline) and exit 2 (cannot tell) both deny; a missing gate script denies. Failing closed is
  deliberate: an unknown baseline is not a baseline.

• Matching is narrow on purpose. An earlier revision matched the bare words experiment and run-eval, which denied
  ls experiments/, read skills/ab-compare/SKILL.md, and git commit -m "experiment run notes". A gate that blocks
  reading the docs teaches people to turn it off. Those are now explicit allow cases in the self-test.

• The SessionStart hook injects the restriction only when no baseline exists, so a lab that is past its baseline sees
  nothing. That is what stops the model from composing hosted-XP-first as a menu option in the first place.

This covers cloned-repo users only. It does not cover the platform Participate flow. A newcomer who arrives through
the Participate button on the game page has no clone, no settings.json, and no hooks, so nothing here can fire for
them. The load-bearing fix for that path is the platform payload itself, and it is a separate PR against
Metta-AI/metta: https://github.com/Metta-AI/metta/pull/20476. The prose mirrors are
https://github.com/Metta-AI/optimizer-agent/pull/6 and https://github.com/Metta-AI/optimizer-skills/pull/10.

Evidence for this half: tools/baseline_pretooluse_gate.sh --self-test passes 16 cases (5 allow, incl. a realistic
payload with fields after tool_input; 9 deny; plus completed-baseline-allows and gate-unavailable-denies). The
sed fallback used when python3 is absent additionally matches the raw payload, so an escaped-quote command such as
bash -c "coworld xp-request ..." cannot slip through a truncated extraction. Both harness JSON files parse, both
scripts are bash -n clean, git diff --check is clean. shellcheck is not available on this machine.

Link to Devin session: https://app.devin.ai/sessions/0c0b9632b5a84c7db7c0abaf26cfd036
Requested by: @arisklar6

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring