P-037 A18-1: STOP — hard cap reached before the population answer

The A18-1 implementation (extending scripts/p037_a18_decompose.py with
--population and --selftest) measures 200 added/changed code lines
relative to #374, against the pre-registered cap of 180. The first draft
was 208; genuine de-duplication brought it to 200 and no further without
golfing or dropping pre-registered checks.

Per §H this is a STOP. Neither --selftest nor the population run was
executed, so there is no result and no conclusion about the population.
The code is committed unrun so the owner can see what the lines are for.
The note §I gives the breakdown and three options:
- raise the cap;
- narrow to per-row L_canonical;
- first add per-row B1 evidence.

ruff and mypy are clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KmiyfrkaG9sJruTcshM2oq

Author: claude

docs/notes/p037-a18-population-decomposition.md

@@ -1,8 +1,10 @@
 # P-037 A18-1: population decomposition (pre-registered)
 
-> Status: **PRE-REGISTERED.** Committed before any implementation. The owner
-> authorized implementing this gate right after this commit. The boundaries
-> below do not move after this SHA.
+> Status: **STOP: HARD CAP REACHED BEFORE THE POPULATION ANSWER (§I).**
+> Pre-registered in `8046688`; the boundaries did not move. The
+> implementation measures 200 added code lines against the 180 cap. Neither
+> `--selftest` nor the population run has been executed. There is no result,
+> and no conclusion about the population.
 >
 > - Base: `3745975`, the head of #374 (A18-0, `RESULT: PASS — A18 8-ROW
 >   DECOMPOSITION HOLDS`, accepted by the owner).
@@ -186,3 +188,53 @@ B1-reconciliation gate needs its own authorization.
 - Not touched: the B1 result, `missing_sidecar`, R, A14, `ConsumesParam`,
   OwnIR, extractor semantics, the application classifier, Phase C, #368, the
   three P-037 classes, production MOS/verdicts, and `own-guarded`.
+
+## I. STOP (MEASURED OBSERVATION): the cap is reached before any answer
+
+§H says that reaching 180 added lines unfinished is a STOP. The
+implementation `scripts/p037_a18_decompose.py` measures **200** counted
+lines added or changed relative to `#374` (`3745975`), counted like A18-0
+(non-blank, non-comment, docstrings included). The first draft measured
+**208**.
+
+The reduction to 200 came from genuine de-duplication only, with no
+formatting or golfing:
+- a shorter docstring;
+- an inlined single-use helper;
+- a dict lookup instead of a quadratic scan;
+- a print that reuses the output dict.
+
+`ruff` and `mypy` are clean. **Neither `--selftest` nor `--population` has
+been run**, because a result obtained over the cap would violate the
+pre-registration. So this STOP says nothing about the population: no row was
+decomposed and no falsifier was exercised.
+
+Where the 200 lines go, roughly:
+
+| part | lines | pre-registered by |
+|---|---|---|
+| forwarding-site enumeration, local-honesty test, document-wide rewrite | ~40 | §C (`L_canonical` document-wide) |
+| forward-closure attribution and the reason rule | ~30 | §C (closure), N4/N5 |
+| typed coverage / drift against B1's per-population counts | ~25 | N1, N2, §B.1 |
+| frozen-tree / per-document plumbing (the A18-0 main still has its own copy) | ~25 | §B |
+| the six falsifiers | ~40 | §E |
+| verdict, cross-tab and evidence writer | ~25 | §F, §G |
+| docstring, imports, small refactors of A18-0 (`ops_at`) | ~15 | |
+
+**Reading (INFERENCE, for the owner).** Going from eight isolated rows to the
+population costs more than the cap assumed, for two reasons:
+- document-wide `L_canonical` needs closure attribution, which A18-0 did not;
+- B1 committed no per-row values for its 28 `EQUAL` rows, so coverage and
+  drift must be reconstructed from class counts.
+
+Neither is a new semantic mechanism, but §H reads the overrun as a signal
+that the abstraction may be wrong, and that is the owner's call. The honest
+options are:
+- raise the cap, knowing what the lines are for;
+- narrow the gate, e.g. per-row `L_canonical` (A18-0's definition), which
+  drops closure attribution but can misclassify callers of normalized
+  wrappers;
+- first make B1 commit per-row evidence (a small B1 evidence addition),
+  which would shrink the coverage/drift part.
+
+Nothing further is done until the owner decides.

scripts/p037_a18_decompose.py

@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
-"""P-037 A18-0: the three-way decomposition of the eight B1 UNEXPLAINED rows.
+"""P-037 A18: the three-way decomposition, A18-0 (the eight B1 UNEXPLAINED rows)
+and A18-1 (`--population`, `--selftest`: docs/notes/p037-a18-population-decomposition.md).
 
 Contract: docs/notes/p037-a18-legacy-decomposition.md (pre-registered in
 ccde27c). For exactly the eight rows committed in B1's evidence, from the
@@ -15,16 +16,21 @@
 witness.
 
 Run:  python scripts/p037_a18_decompose.py --out docs/evidence/p037-a18/eight-rows.json
+      python scripts/p037_a18_decompose.py --selftest
+      python scripts/p037_a18_decompose.py --population --out docs/evidence/p037-a18/population.json
 """
 from __future__ import annotations
 
 import argparse
+import collections
+import contextlib
 import copy
 import json
 import re
 import subprocess
 import sys
 import tempfile
+from collections.abc import Iterator
 from pathlib import Path
 from typing import Any
 
@@ -61,6 +67,11 @@ def names(op: Any, p: str) -> bool:
     return p in (op.get("var"), op.get("src"), op.get("result")) or p in (op.get("args") or [])
 
 
+def ops_at(fn: Any, line: Any, p: str) -> list[tuple[list[Any], int]]:
+    return [(lst, i) for lst, i in op_sites(fn["body"])
+            if lst[i].get("line") == line and names(lst[i], p)]
+
+
 def locate(facts: Any, row: Any) -> tuple[Any, Any, list[tuple[list[Any], int]]]:
     """The row's function, its one forwarding sidecar call, and the legacy ops on
     the parameter at that call's statement line."""
@@ -69,10 +80,7 @@ def locate(facts: Any, row: Any) -> tuple[Any, Any, list[tuple[list[Any], int]]]
              if any(a["kind"] == "param" and a.get("source_param") == row["ordinal"]
                     for a in c["args"])]
     call = calls[0] if len(calls) == 1 else None
-    line = call["statement_line"] if call else None
-    ops = [(lst, i) for lst, i in op_sites(fn["body"])
-           if lst[i].get("line") == line and names(lst[i], row["param"])]
-    return fn, call, ops
+    return fn, call, ops_at(fn, call["statement_line"] if call else None, row["param"])
 
 
 def decompose(facts: Any, rep: Any, row: Any) -> dict[str, Any]:
@@ -115,7 +123,8 @@ def shape_probe(tree: Path, files: list[str], row: Any, line: int, work: Path) -
         facts, rep, _ = b1.run_document(tree, files, work)
     finally:
         src.write_text(text)
-    ops = locate(facts, row)[2]
+    fn = next(f for f in facts["functions"] if f["name"] == row["method"])
+    ops = ops_at(fn, line, row["param"])
     return {"line_before": old.strip(), "line_after": lines[line - 1].strip(),
             "op_after": ops[0][0][ops[0][1]]["op"] if len(ops) == 1 else None,
             "actual_after": coord(rep, row["method"], "index", row["index"])["legacy"]}
@@ -132,15 +141,221 @@ def witnessed(d: dict[str, Any], row: dict[str, Any]) -> bool:
     return False
 
 
+# ---------------------------------------------------------------------------
+# A18-1: the population gate
+# ---------------------------------------------------------------------------
+
+LOCAL = {"release": "must", "use": "no"}  # a legacy op's own local value
+
+
+def reason_of(actual: str, canonical: str, mechs: set[str], broken: bool) -> str:
+    """The normalization column: only the pre-registered reasons."""
+    if broken:
+        return "UNEXPLAINED"
+    if canonical == actual:
+        return "EQUAL"
+    if len(mechs) == 1 and mechs <= {"CONSUMES_PARAM_FOLD", "ARGUMENT_SHAPE_LOSS"}:
+        return next(iter(mechs))
+    return "UNEXPLAINED"
+
+
+def population(facts: Any, rep: Any, rows: list[dict[str, Any]], probe: Any,
+               rewrite: bool = True) -> list[dict[str, Any]]:
+    """One document: normalize every locally non-honest forwarding site of every
+    comparable row, run the production MOS once, attribute through closures."""
+    canon = copy.deepcopy(facts)
+    fns = {f["name"]: f for f in canon["functions"]}
+    info: dict[tuple[str, int], dict[str, Any]] = {}
+    for r in rows:
+        entry: dict[str, Any] = {"broken": False, "sites": []}
+        fn = fns[r["method"]]
+        for c in (fn.get("guarded_facts") or {}).get("calls", []):
+            slots = [a["param"] for a in c["args"]
+                     if a["kind"] == "param" and a.get("source_param") == r["ordinal"]]
+            if not slots:
+                continue
+            ops, callee = ops_at(fn, c["statement_line"], r["param"]), coord(
+                rep, c["callee"], "ordinal", slots[0])
+            if len(ops) != 1 or callee is None:
+                entry["broken"] = True
+                continue
+            lst, i = ops[0]
+            kind, mech = lst[i]["op"], None
+            if LOCAL.get(kind, callee["legacy"]) != callee["legacy"]:
+                mech = ("CONSUMES_PARAM_FOLD" if kind == "release" else
+                        "ARGUMENT_SHAPE_LOSS" if probe(r, c["statement_line"]) else "UNKNOWN")
+                if rewrite:  # the A18-0 honest positional forward
+                    lst[i] = {"op": "call", "callee": c["callee"], "sig": c["sig"], "line": c[
+                        "statement_line"], "args": ["_"] * callee["index"] + [r["param"]]}
+            entry["sites"].append({"callee": (c["callee"], callee["index"]), "op": kind,
+                                   "line": c["statement_line"], "mechanism": mech})
+        info[(r["method"], r["index"])] = entry
+    after_rep = report(canon)
+    out = []
+    for r in rows:
+        k = (r["method"], r["index"])
+        seen, stack, mechs = {k}, [k], set()
+        while stack:
+            for site in info.get(stack.pop(), {"sites": []})["sites"]:
+                mechs |= {site["mechanism"]} - {None}
+                if site["callee"] not in seen:
+                    seen.add(site["callee"])
+                    stack.append(site["callee"])
+        after = coord(after_rep, r["method"], "index", r["index"])
+        reason = reason_of(r["actual"], after["legacy"], mechs, info[k]["broken"])
+        out.append({**r, "canonical": after["legacy"], "guarded_after": after["guarded"],
+                    "semantic_class": after["class"], "normalization_reason": reason,
+                    "closure_mechanisms": sorted(mechs), "sites": info[k]["sites"],
+                    "equal_kind": (("identity" if not mechs else "after_rewrite")
+                                   if reason == "EQUAL" else None)})
+    return out
+
+
+def coverage(rows: list[dict[str, Any]], expected: dict[str, dict[str, int]],
+             committed: list[dict[str, Any]]) -> list[tuple[str, str]]:
+    """N1 (missing/duplicate) and N2 (drift) against B1's committed evidence."""
+    bad = []
+    keys = [(r["doc"], r["method"], r["index"]) for r in rows]
+    if len(set(keys)) != len(keys):
+        bad.append(("duplicate", "a comparable row occurs twice"))
+    got: dict[str, collections.Counter[str]] = collections.defaultdict(collections.Counter)
+    for r in rows:
+        got[r["pop"]][r["b1_class"]] += 1
+    for pop in set(expected) | set(got):
+        want = {c: n for c, n in expected.get(pop, {}).items() if c != "NO_GUARDED_EVIDENCE"}
+        if dict(got[pop]) != want:
+            kind = "drift" if sum(got[pop].values()) == sum(want.values()) else "missing"
+            bad.append((kind, f"{pop}: {dict(got[pop])} != B1 {want}"))
+    index = {(r["doc"], r["method"], r["index"]): r for r in rows}
+    for u in committed:
+        hit = index.get((u["doc"], u["method"], u["index"]))
+        if hit is None:
+            bad.append(("missing", f"committed row absent: {u['doc']} {u['method']}"))
+        elif (hit["actual"], hit["guarded"]) != (u["legacy"], u["guarded"]):
+            bad.append(("drift", f"committed row drifted: {u['doc']} {u['method']}"))
+    return bad
+
+
+@contextlib.contextmanager
+def frozen_tree() -> Iterator[tuple[Path, Path, dict[str, tuple[str, list[str]]]]]:
+    commit = b1.frozen(b1.POPULATION_COMMIT)
+    with tempfile.TemporaryDirectory(prefix="p037-a18-") as td:
+        tree = Path(td) / "tree"
+        tree.mkdir()
+        archive = subprocess.run(["git", "archive", commit], cwd=ROOT, capture_output=True,
+                                 check=True).stdout
+        subprocess.run(["tar", "-x", "-C", str(tree)], input=archive, check=True)
+        yield tree, Path(td), {n: (pop, f) for pop, n, f in b1.documents(tree, commit)}
+
+
+def comparable(tree: Path, work: Path, name: str, pop: str, files: list[str]
+               ) -> tuple[Any, Any, list[dict[str, Any]], Any]:
+    facts, rep, _ = b1.run_document(tree, files, work)
+    rows = [{"doc": name, "pop": pop, "method": s["method"], "index": s["index"],
+             "ordinal": s["ordinal"], "param": s["param"], "actual": s["legacy"],
+             "guarded": s["guarded"], "b1_class": s["class"]}
+            for s in rep["summary"] if s["class"] != "NO_GUARDED_EVIDENCE"]
+
+    def probe(r: Any, line: int) -> bool:
+        p = shape_probe(tree, files, r, line, work)
+        return bool(p["op_after"] == "release" and p["line_after"] != p["line_before"])
+    return facts, rep, rows, probe
+
+
+def selftest() -> int:
+    fails = 0
+
+    def check(label: str, ok: bool) -> None:
+        nonlocal fails
+        print(f"{'ok ' if ok else 'MISS'} {label}")
+        fails += not ok
+    with frozen_tree() as (tree, work, docs):
+        facts, rep, rows, probe = comparable(tree, work, "guard-forward-bare",
+                                             *docs["guard-forward-bare"])
+        outer = [r for r in rows if r["method"] == "ShapeForwardBare.Outer"]
+        on = population(facts, rep, outer, probe)[0]["normalization_reason"]
+        off = population(facts, rep, outer, probe, rewrite=False)[0]["normalization_reason"]
+        check(f"F1  a fold row without its rewrite stops being a fold ({on} -> {off})",
+              on == "CONSUMES_PARAM_FOLD" and off != on)
+        facts, rep, rows, probe = comparable(tree, work, "arg-cast-and-bang",
+                                             *docs["arg-cast-and-bang"])
+        cast = [r for r in rows if r["method"] == "ShapeCastBang.Cast"]
+        broken = copy.deepcopy(facts)
+        fn = next(f for f in broken["functions"] if f["name"] == "ShapeCastBang.Cast")
+        for c in fn["guarded_facts"]["calls"]:
+            c["statement_line"] += 100
+        got = population(broken, rep, cast, probe)[0]["normalization_reason"]
+        check(f"F2  a broken sidecar-to-forward link is UNEXPLAINED, not a fold ({got})",
+              got == "UNEXPLAINED")
+        check("F3  actual=unknown, canonical=must, no mechanism is UNEXPLAINED",
+              reason_of("unknown", "must", set(), False) == "UNEXPLAINED")
+        sem = population(facts, rep, cast, probe, rewrite=False)[0]["semantic_class"]
+        check(f"F4  G=may vs L_canonical=no is semantic UNEXPLAINED ({sem})",
+              sem == "UNEXPLAINED")
+        base = population(facts, rep, rows, probe)
+        want = {"p037-shapes": dict(collections.Counter(r["b1_class"] for r in base))}
+        check("F5' the coverage check passes on a complete set", not coverage(base, want, []))
+        check("F5  a deleted comparable row fails coverage", bool(coverage(base[1:], want, [])))
+        check("F6  a duplicated comparable row fails coverage",
+              bool(coverage([*base, base[0]], want, [])))
+    print(f"RESULT: {'all falsifiers fire' if not fails else f'{fails} falsifier(s) missed'}")
+    return 1 if fails else 0
+
+
+def population_run(out_path: str) -> int:
+    ev = json.loads(EVIDENCE.read_text())
+    expected = {k.split("/", 1)[1]: v for k, v in ev["counts"].items()
+                if k.startswith("summary/")}
+    committed = [u for u in ev["unexplained"] if u["level"] == "summary"]
+    rows: list[dict[str, Any]] = []
+    with frozen_tree() as (tree, work, docs):
+        for name, (pop, files) in docs.items():
+            facts, rep, comp, probe = comparable(tree, work, name, pop, files)
+            if comp:
+                rows += population(facts, rep, comp, probe)
+    sem = collections.Counter(r["semantic_class"] for r in rows)
+    norm = collections.Counter(r["normalization_reason"] for r in rows)
+    cross = collections.Counter(f"{r['semantic_class']} x {r['normalization_reason']}"
+                                for r in rows)
+    cov = coverage(rows, expected, committed)
+    drift = [msg for kind, msg in cov if kind == "drift"]
+    moved = [r for r in rows if r["guarded_after"] != r["guarded"]]
+    verdict = ("VOID — B1 instrument drift" if drift else
+               "FAIL — A18-1 COVERAGE BROKEN" if cov or moved else
+               "FAIL — A18-1 NORMALIZATION MODEL INCOMPLETE" if norm["UNEXPLAINED"] else
+               "FAIL — A18-1 SEMANTIC UNEXPLAINED" if sem["UNEXPLAINED"] else
+               "PASS — A18 POPULATION DECOMPOSITION HOLDS")
+    out = {"schema": "p037-a18-population/1", "population_commit": b1.frozen(b1.POPULATION_COMMIT),
+           "instrument_commit": b1.git("rev-parse", "HEAD").strip(),
+           "comparable_expected": sum(n for v in expected.values() for c, n in v.items()
+                                      if c != "NO_GUARDED_EVIDENCE"),
+           "covered": len(rows), "coverage_problems": cov, "guarded_moved": len(moved),
+           "semantic": dict(sem), "normalization": dict(norm),
+           "equal_kind": dict(collections.Counter(r["equal_kind"] for r in rows
+                                                  if r["equal_kind"])),
+           "cross_tab": dict(cross), "rows": rows, "result": verdict}
+    Path(out_path).parent.mkdir(parents=True, exist_ok=True)
+    Path(out_path).write_text(json.dumps(out, indent=1) + "\n", encoding="utf-8")
+    print(json.dumps({k: v for k, v in out.items() if k != "rows"}, indent=1))
+    print(f"RESULT: {verdict}")
+    return 0 if verdict.startswith("PASS") else 1
+
+
 def main(argv: list[str]) -> int:
     ap = argparse.ArgumentParser(description=__doc__,
                                  formatter_class=argparse.RawDescriptionHelpFormatter)
-    ap.add_argument("--out", required=True)
+    ap.add_argument("--out")
+    ap.add_argument("--population", action="store_true")
+    ap.add_argument("--selftest", action="store_true")
     args = ap.parse_args(argv)
+    if args.selftest:
+        return selftest()
     commit = b1.frozen(b1.POPULATION_COMMIT)
-    if b1.git("status", "--porcelain", "--untracked-files=no").strip():
-        print("REFUSED: the instrument tree is dirty; commit it first")
+    if b1.git("status", "--porcelain", "--untracked-files=no").strip() or not args.out:
+        print("REFUSED: needs --out and a clean instrument tree (commit it first)")
         return 2
+    if args.population:
+        return population_run(args.out)
     b1_rows = [u for u in json.loads(EVIDENCE.read_text())["unexplained"]
                if u["level"] == "summary"]
     rows: list[dict[str, Any]] = []