polyjit

Do machine-generated agent harnesses preserve hard constraints?

This repository measures constraint survival through JIT-Agent's released harness-generation pipeline, and then through five rounds of context compaction using the measurement protocol of The Compaction Cliff in Long-Running AI Agent Memory (Zerhoudi, Mitrović & Granitzer, CIKM '26).

Headline result: 18/18 generated harnesses are structurally valid; 1.7% of injected hard constraints survive generation (95% CI 0–4.4%). Of the constraints that reach the runtime context, 21.7% survive one compaction round and 0% survive two — against 52.8% → 10.1% over five rounds on hand-written configurations. Each harness's own generated summary prompt erases 100% of constraints in a single round.

Full write-up: REPORT.md · pre-registered design: PROTOCOL.md

Layout

PROTOCOL.md            pre-registered experimental protocol
REPORT.md              results, mechanism, threats to validity
experiments/
  build_inputs.py      constraints.json + specs.json (deterministic, seed 11)
  jitgen.py            faithful re-render of JIT-Agent's generation prompt (desc mode)
  llm.py               LLM endpoint wrapper (claude -p)
  run_m1.py            M1: generation survival (18 harnesses)
  run_m2.py            M2: 5-round compaction survival (3 compactor conditions)
  analyze.py           aggregation + bootstrap CIs -> results/summary.json
  knowledge_triage/    vendored deterministic subset of the Passau reference
                       implementation (classifier, key-token test, TypeCompact) — Apache-2.0
  results/             every generated harness, every compaction round, summary.json

Reproduce

Requires Python 3.11+, pyyaml, jinja2, and the Claude Code CLI on PATH (LLM calls run through claude -p; ~78 Sonnet-class calls total). Clone the two upstream repos next to experiments/ (they are intentionally not vendored):

git clone https://github.com/bingreeky/JIT
git clone https://github.com/searchsim-org/cikm26-knowledge-triage

cd experiments
python build_inputs.py      # inputs (seed 11; committed copies included)
python run_m1.py --workers 1
python run_m2.py --workers 1
python analyze.py           # -> results/summary.json

Measured 2026-08-31 → 09-01 with claude-sonnet-4-6 as the meta model via JIT-Agent's documented hosted-API mode (see REPORT.md, Deviations).

Sources

• JIT-Agent: arXiv:2608.25593 · bingreeky/JIT · weights
• Compaction Cliff: arXiv:2608.22752 · searchsim-org/cikm26-knowledge-triage · AgentArtifactCorpus
• Related: arXiv:2606.22528 (survival→violation link) · arXiv:2604.20911 (omission vs commission decay)

License

MIT for the code in this repository, except experiments/knowledge_triage/, which is vendored from the Passau reference implementation under Apache-2.0 (see its module headers).