research-log

Research Log

This directory is the public devlog of DM-Code-Agent's algorithm-track upgrade.

It is not a tutorial. It is a chronological record of:

• the problems we noticed,
• the algorithms or designs we picked to address them,
• the experiments we ran (with seeds, configs, and ablations),
• the things that did not work, and
• the next bets.

The intent is to make the agent's evolution inspectable, the same way JSONL traces make a single run inspectable. If you want to know why the project looks the way it does, the answer should be reachable from one of these entries.

Index

#	Title	Phase	Status
00	Why this upgrade — motivation, goals, constraints	P0	active
01	SWE-bench Lite harness, sampling, and the road to numbers	P1	active
02	Reflexion: episodic memory across trials	P2	implementation landed; ablation pending
04	Critic agent and self-consistency selection	P4	implementation landed; ablation pending
05	Adaptive replanning and token economics	P5	implementation landed; real cross-model eval frozen
06	Final write-up: what worked, what didn't, what's next	P6	published
07	Trace diff for offline behavioral comparison	post-v2	implementation landed
08	Trace analyzer for failure attribution and verification gaps	post-v2	implementation landed
09	More realistic maintenance benchmark tasks	post-v2	implementation landed
10	Benchmark confidence intervals for repeated samples	post-v2	implementation landed
11	Self-consistency uncertainty metadata	post-v2	implementation landed
12	Adaptive replanning repeated-failure signals	post-v2	implementation landed
13	Benchmark manifest provenance and suite signatures	post-v2	implementation landed
14	Benchmark trace-analysis metadata	post-v2	implementation landed
15	Trace directory analysis aggregation	post-v2	implementation landed
16	Confidence-aware token economics reports	post-v2	implementation landed
17	Packaging and CI maintenance benchmark task	post-v2	implementation landed
18	Patch-fingerprint self-consistency voting	post-v2	implementation landed
19	Benchmark manifest diff CLI	post-v2	implementation landed
20	Economics manifest guard	post-v2	implementation landed
21	Trace-analysis Markdown reports	post-v2	implementation landed
22	Repeated-failure policy experiment	post-v2	implementation landed
Distribution	Launch checklist and external posting plan	P6	local checklist
Interview	Resume/interview talking points	P6	private prep notes

Conventions

• Each entry is a single Markdown file NN-slug.md, ordered by date.
• Numbers are stable. Do not renumber after publication.
• Each entry begins with a short ## TL;DR, then ## Context, the work, ablation tables, and ## Open questions / next bets.
• Ablation tables embed enough to reproduce: model, provider, dataset slice, seed, configuration flags, raw bench_reports/*.json path.
• Failures are first-class content. Negative results are kept in.

Reproducing

Every ablation table in this log links to a JSON in bench_reports/. To reproduce, follow the command shown in the entry; the numbers should match within sampling noise. If they do not, please open an issue and tag the entry.