Skip to content

README.md

Latest commit

 

History

History
91 lines (69 loc) · 3.49 KB

README.md

File metadata and controls

91 lines (69 loc) · 3.49 KB
Raw

Experiment reproduction pipeline

This directory contains the pipeline for reproducing the experiments, result tables, and generated figures of the accompanying article.

  • jobs/: generators that write the simulation and AIHL evaluation jobs (YAML configs, sbatch scripts, launchers, manifests) for the experiment sweep.
  • tables/: scripts that turn raw run data and AIHL outputs into the result tables behind every figure.
  • figures/: scripts that turn those tables into the figures.
  • results/: default output location for the artifacts of record (results/tables/ for result tables and LaTeX tables, results/figures/ for the figures built from them).
  • generated/: output area for the generated job batches.

Reproduction steps

Stage 1: generate and run the simulations (Slurm cluster-based)

# 14 families x conditions x 5 seeds -> data/simulations/
python -m exploratory_experiments.jobs.generate_simulations

# Submit, then collect simulator metrics when finished:
bash exploratory_experiments/generated/jobs/simulations/scripts/launcher.sh
bash exploratory_experiments/generated/jobs/simulations/scripts/collect_results.sh

Stage 2: generate and run the AIHL evaluations (LLM API calls)

Requires GEMINI_API_KEY and ANTHROPIC_API_KEY; each evaluation is an assessor/critic dialogue (gemini-2.5-flash assessor, claude-sonnet-4 critic). Cost is estimated per run by the collection step using tables/model_pricing.json.

# 150 evaluations (3 random distinct seeds per condition) + 18 standalone
# per-component population-split evaluations -> data/aihl/
python -m exploratory_experiments.jobs.generate_aihl \
  --manifest exploratory_experiments/generated/jobs/simulations/manifest.jsonl \
  --batch-dir exploratory_experiments/generated/jobs/aihl \
  --population-split-components

# Sequential launchers (safe to run repeatedly, completed jobs are skipped):
bash exploratory_experiments/generated/jobs/aihl/scripts/launcher.sh
bash exploratory_experiments/generated/jobs/aihl/scripts/components_launcher.sh

Stage 3: build the result tables

conda run -n language_evolution bash exploratory_experiments/build_tables.sh

Tables land in results/tables/ (override with TABLES_ROOT=). Stage 3 reads the job manifests from Stages 1 and 2. If the run data is already present (for example on the machine where the experiments were run), generate the manifests without submitting anything:

python -m exploratory_experiments.jobs.generate_simulations
python -m exploratory_experiments.jobs.generate_aihl \
  --manifest exploratory_experiments/generated/jobs/simulations/manifest.jsonl \
  --batch-dir exploratory_experiments/generated/jobs/aihl \
  --population-split-components \
  --seeds-from-existing

AIHL tables are skipped automatically if no AIHL manifest is present.

Stage 4: build the figures and LaTeX tables

conda run -n language_evolution bash exploratory_experiments/build_figures.sh

build_figures.sh reads results/tables/ and writes results/figures/, with the LaTeX tables placed next to their source TSVs (override with TABLES_ROOT= and OUTPUT_ROOT=).

Notes

  • Population split and cascade targeting are part of the sweep but reported standalone. The per-component population-split AIHL scores come from a dedicated job set (--population-split-components).
  • Simulator configs enable a population cache under data/_cache_populations, so all conditions of a family share identical initial populations per seed.