docs: add 2026 H2 improvement plan for Creature Lab - #16
Merged
Conversation
Add a research-grounded improvement plan that keeps Creature Lab distinct from Agentarium. Prioritizes upgrading the run report (self-contained HTML run card, before/after comparison, reproducibility block, integrated zoo gallery) followed by terrain library, robustness/sim2sim reporting, quality-diversity gallery, a sharpened offline-first LLM design loop, and packaging/trust work. Grounded in a scan of EvoGym, Revolve2, RoboMorph, and reproducibility-card conventions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ac8LrapgJhth67XouC6QtB
…Phase R) Implement Phase R of the 2026 improvement plan: the report is now the highest-fidelity artifact in the loop. - creature_lab/reports_html.py: report_to_html renders a single offline HTML file with a score-breakdown bar, signal sparklines (CoM height, score, joint energy), a top-down root-path SVG plot, diagnosis, warnings, improvement lineage, and reproducibility info, plus an optional embedded GIF preview (data: URI, no network). comparison_to_html overlays two runs' root paths and renders a signed signal-delta table. - reports.py: build_report now includes a reproducibility block (schema/lab version, timestep, seed, hashes, backend, and a runnable reproduce command), also rendered into the Markdown report. Added build_report_bundle (report + raw trace/creature for HTML renderers) and build_comparison (score/signal deltas between two reports). - CLI: `report --html`, `compare --html` (writes a diff report instead of opening Viser), and `gallery build --zoo` now also emits index.html with baseline-vs-current score coloring per creature. - 11 new tests covering HTML rendering, no-creature/no-media degradation, comparison deltas, and CLI wiring; verified no external URLs are ever embedded. Full suite green (252 passing), ruff clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ac8LrapgJhth67XouC6QtB
Implement Phase 1 of the 2026 improvement plan: slope/steps/gaps/rough terrain, deterministic and identical in shape on both simulator backends. - TaskSpec.terrain gains a TerrainType enum (slope/steps/gaps/rough) and the per-type parameters (slope_angle, step_height/length, gap_width/period, roughness/seed), with a gap_width < gap_period cross-check. - creature_lab/terrain.py: pure, backend-agnostic heightfield generation (heightfield_grid, heightfield_range, normalized_heightfield_data, flatten_for_heightfield_api) so PyBullet and MuJoCo build the same ground shape from the same TaskSpec. gaps terrain keeps a solid platform around the origin so a creature always spawns on ground regardless of gap phase. - PyBulletBackend builds a GEOM_HEIGHTFIELD body; MuJoCoBackend builds a <hfield> asset (export/mjcf.py) and fills model.hfield_data at runtime. Both engines turned out to auto-recenter/reindex heightfield data in ways their docs don't state plainly, and share the same axis convention - verified empirically (a PyBullet raycast probe, a MuJoCo free-falling probe body) before trusting the physics, rather than assuming it worked from the first pass that ran without crashing. - Three new quadruped zoo tasks (slope_climb, step_over, gap_cross) with measured baselines. - 32 new tests across terrain math, both backends, MJCF export, and the zoo. Full suite green (284 passing), ruff clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ac8LrapgJhth67XouC6QtB
Implement Phase 2 of the 2026 improvement plan: measure the "specs are portable, physics is backend-dependent" contract instead of just stating it. - creature_lab/robustness.py: pure perturbation + trial-statistics module, following evolve.py's callback-injection pattern (never touches physics directly). Deterministically jitters part masses and terrain friction under a seeded RNG and re-validates the result. - CLI: `robustness runs/<id> --trials N` re-simulates under small seeded perturbations and reports score mean/std/min/max and fail rate - reveals gaits that only work for the exact recorded parameters. - CLI: `sim2sim runs/<id>` runs the same creature/task on PyBullet and MuJoCo and reports the score gap and mean root-position divergence (reusing viewers/overlays.root_path). Running it on the packaged quadruped surfaced a real gap (PyBullet 0.57 vs. MuJoCo ~0.00 for the same open-loop gait) - the kind of result this phase exists to surface. - Both commands take --save to write a reportable run; build_report and both report renderers (Markdown, HTML) grow an optional Robustness/Sim2Sim section when robustness.json/sim2sim.json are present. - 20 new tests (pure perturbation/statistics math + CLI smoke tests, gated on the sim/mujoco extras). Full suite green (296 passing), ruff clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ac8LrapgJhth67XouC6QtB
Implement Phase 3 of the 2026 improvement plan: a QD search's most compelling output - the diversity of solutions - was previously invisible. - evolve --strategy map_elites now persists each filled cell's CreatureSpec into archive.json (previously only score/features were saved, discovered while implementing archive export - there was nothing to export yet). - CLI: `archive show <run>` prints filled cells as a ranked table; --html renders a red-to-green scored heatmap (creature_lab.reports_html. archive_to_html); --html --task <task.json> additionally renders a replay GIF per cell (reusing render_trace/write_animation). - CLI: `archive export <run> --cell row,col --out spec.json` pulls one elite out as a standalone, editable CreatureSpec - feeds back into scaffold/ask/evolve like any other spec. Verified round-trip on a real 60-attempt map_elites run. - 8 new tests (heatmap rendering, empty-archive handling, GIF embedding, CLI show/export, unknown-cell error). Full suite green (301 passing), ruff clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ac8LrapgJhth67XouC6QtB
…olve strategy (Phase 4) Implement Phase 4 of the 2026 improvement plan, offline-first throughout. - agents/loop.py: Observation gains a `diagnosis` field; design_loop gains an optional `diagnose` callback invoked on the current best before every proposal. agents/prompts.py includes "diagnosed issues: ..." in the prompt whenever a pattern is detected. - CLI: `ask` wires a diagnose_fn (reusing diagnosis.diagnose) into design_loop, so both the online LLM and the offline policy's Observation see *why* the creature is scoring the way it is, not just the score. - evolve.py: new `llm_mutate` mutate_fn that proposes one edit through the same offline RandomToolPolicy + validated tool layer used by `ask`, reseeded per-call from the caller's rng so --seed stays reproducible. CLI: `evolve --strategy llm` dispatches to hill_climb with this operator; each attempt's rationale is saved into lineage.json's node `note` field (shown by `creature-lab lineage`) since that's the artifact this command already produces, rather than agent.json (which belongs to `ask`). - Verified end-to-end: --strategy llm improved the packaged quadruped from 0.57 to 0.86 in 10 attempts and reproduces exactly for a fixed seed. - 20 new tests across the design loop, prompts, evolve mutator, and CLI. Full suite green (311 passing), ruff clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ac8LrapgJhth67XouC6QtB
Implement Phase 5 of the 2026 improvement plan, minus the one step that needs a human: publishing to PyPI requires real maintainer credentials, so that step is left for a maintainer to run (uv build && uv publish) rather than attempted here. Verified rather than assumed: - Built the wheel and diffed its contents against the source tree - all zoo creatures/tasks/baselines (including the new terrain tasks) package correctly. - Installed the wheel into a clean venv outside the checkout and ran `creature-lab zoo run quadruped --task slope_climb` from an unrelated directory: the score matched the committed baseline exactly, confirming packaged data actually works standalone (not just "should work"). Added: - A real, measured MuJoCo baseline for every packaged zoo creature/task pair (baselines/<task>.mujoco.json, 17 total) - previously only PyBullet baselines existed. zoo_baseline() takes an optional `backend` param to pick the right file. - Found and fixed a real bug this surfaced: `bench --zoo --backend mujoco` was comparing MuJoCo scores against the PyBullet baseline regardless of --backend, so the pass/fail threshold was silently wrong for every non-default-backend benchmark run. - CHANGELOG.md, linked from README.md and pyproject.toml; docs/ROADMAP.md updated to reflect what's done vs. what needs a maintainer. - 6 new tests. Full suite green (314 passing), ruff clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ac8LrapgJhth67XouC6QtB
All of Phase R and Phases 1-5 landed this session; update the sequencing table with final status instead of the original dependency-ordering guess. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ac8LrapgJhth67XouC6QtB
…eport (Phase A)
Fixes the highest-priority gap from the post-implementation review: replaying
or exporting a run on non-flat terrain (slope/steps/gaps/rough) showed the
creature floating above or sinking into a flat floor, even though the
recorded poses were correct — only the ground was misrepresented.
- creature_lab/terrain.py: new height_at() — nearest-cell terrain height
lookup for visual overlays (not precise enough for physics, which is why
the backends use the full grid instead).
- pybullet_backend.render_trace() takes an optional `task` and reuses the
existing _build_ground() helper to draw the real heightfield shape
instead of always loadURDF("plane.urdf"). Verified with a real pixel-diff
test: >50% of pixels differ between a flat and a gaps-terrain render of
the same static scene.
- viewers/viser_viewer.py: build_scene() draws a trimesh heightfield mesh
for non-flat terrain (a flat grid otherwise); add_debug_overlays() places
the root-path dots at the actual terrain height via height_at() instead
of a flat z=0.005, verified against the slope formula in a headless-viser
test.
- Threaded `task`/`task_spec` through every render_trace call site (export,
gallery build, archive show --task) and into report_to_html (the run
report's embedded GIF), so a run's terrain is drawn everywhere it's
visualized, not just simulated.
- 11 new tests: height_at math, render_trace terrain pixel-diff, headless
viser floor-mesh-vs-grid and root-path-height assertions, and a CLI
wiring smoke test. Full suite green (325 passing), ruff clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ac8LrapgJhth67XouC6QtB
…(Phase B) Closes finding #3 from the post-implementation review: a report or `inspect` gave no indication of which terrain a run used, even though the terrain library (Phase 1) had been shipped for a while. - creature_lab/terrain.py: new describe_terrain() — one-line human summary per terrain type (e.g. "slope (angle=0.2 rad, friction=1)"), tested against every terrain type. - `inspect` (table and --json) and every report renderer (build_report, report_to_markdown, report_to_html) now show the terrain. - docs/KNOWN_ISSUES.md: a living table for latent gaps and deliberate limitations found in review (diagnosis terrain-awareness, the finite heightfield extent, the rows==cols axis-mapping assumption, and CI not wheel-testing) plus the accepted-limitations this repo already lives with (backend-dependent physics, gaps-as-deep-pit) — so findings have a home instead of being re-discovered or lost between sessions. - 8 new tests. Full suite green (333 passing), ruff clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ac8LrapgJhth67XouC6QtB
…s (Phase C) Closes findings #4, #6, and part of #7 from the post-implementation review. - archive show --html --task now embeds per-cell replay GIFs as data: URIs (matching the run report's embedded GIF) instead of writing them to a sibling directory the page silently breaks without if moved. - CI builds the wheel, installs it into a fresh venv, and runs doctor + zoo run against the installed package (not the checkout) on every push/PR - verified locally first (reproduces the exact committed baseline score, 0.5736, from a clean install). Previously this was only checked once by hand in Phase 5 and could regress silently. - validate_episode_inputs warns when a task's target lies outside the generated terrain's finite extent on non-flat terrain - a concrete, checkable case of the "finite heightfield extent" known issue. Target-less locomotion tasks can't be checked this way (no way to derive expected travel from the spec alone), so that part stays open and documented. - terrain.py asserts DEFAULT_ROWS == DEFAULT_COLS at import time: the PyBullet/MuJoCo heightfield axis convention was only verified empirically for a square grid, so changing the constants to a non-square shape now fails loudly instead of silently swapping which world axis the terrain varies along. - docs/KNOWN_ISSUES.md and CHANGELOG.md updated to reflect what's closed vs. still open. - 5 new tests. Full suite green (338 passing), ruff clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ac8LrapgJhth67XouC6QtB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a research-grounded improvement plan that keeps Creature Lab distinct
from Agentarium. Prioritizes upgrading the run report (self-contained HTML
run card, before/after comparison, reproducibility block, integrated zoo
gallery) followed by terrain library, robustness/sim2sim reporting,
quality-diversity gallery, a sharpened offline-first LLM design loop, and
packaging/trust work. Grounded in a scan of EvoGym, Revolve2, RoboMorph,
and reproducibility-card conventions.