Static benchmarks ask what a model knows. ChemWorld asks how it experiments when the answer is hidden.
ChemWorld is a replayable causal physical-chemistry environment for training and evaluating experimental agents. It separates a physical causal world substrate, an experimental interaction runtime, and task/evaluation contracts. Under partial observability, finite budgets, and operational constraints, agents choose interventions and measurements, form hypotheses, and revise their strategies from feedback generated by executable mechanisms.
ChemWorld provides the environment and evaluation contracts; it does not retrain a hosted Agent during an evaluation campaign. Agent intelligence may live in externally trained weights, while within-campaign adaptation is measured through memory, belief, and action updates.
The same public task can run across different hidden kinetic, phase-behavior, or process worlds. This makes it possible to test whether an agent identifies and adapts to changing rules instead of memorizing one optimal recipe. Bayesian optimization, reinforcement learning, world models, LLM tool agents, and humans share the same versioned interaction and replay contracts.
ChemWorld does not claim to predict arbitrary real reactions. Its first goal is to make experimental decision making scalable, comparable, and falsifiable; its long-term bridge question is whether virtual training reduces the number, risk, and cost of experiments needed to adapt to independent models, real data, and narrow physical systems.
Website: https://sunyrain.github.io/ChemWorld/
| Layer | Role |
|---|---|
| Physical Causal World Substrate | Typed state, executable physical laws, instruments, constraints, and controlled world shifts |
| Experimental Interaction Runtime | Operations, transactions, measurements, lifecycle, resource ledgers, and trajectories |
| Task and Evaluation Contract | Public objectives, information/action permissions, budgets, scoring, and scenario distributions |
| ChemWorld Interface | Optional Student and Agent Observatory views over the same environment API |
| ChemWorld Bridge | A validation roadmap for independent backends, real datasets, and physical systems |
The agent, trainer, model weights, and private agent memory sit outside the three environment layers. Core,
Diagnostic, and Extended refer only to evaluation-suite roles, not separate engines.
ChemWorld supports Python 3.11 and 3.12.
git clone https://github.com/sunyrain/ChemWorld.git
cd ChemWorld
python -m pip install -e ".[dev]"The commands above are the general editable-install route for users. Repository contributors should
use the committed lockfile: run uv sync --extra dev once, then execute checks and repository tools
with uv run --no-sync ...; see CONTRIBUTING.md.
The default development extra collects the complete suite and skips optional RL modules when their
backend is absent. RL development and training use python -m pip install -e ".[dev,rl]", which also
installs the supported Stable-Baselines3/Torch stack.
Run and independently verify a complete trajectory:
chemworld tasks list
chemworld run --task reaction-to-assay --agent random --seed 0
chemworld verify --constitution --submission runs/<trajectory>.jsonl
chemworld evaluate --submission runs/<trajectory>.jsonlEvaluation replays the trajectory and recomputes metrics rather than trusting a score supplied by the agent.
Browse existing experiments and replay local files in the Experiment Observatory:
uv run --no-sync python -m apps.experiment_explorer
# Open http://127.0.0.1:8890The local, read-only interface indexes current bound reports and run archives, plots recorded metrics, and supports JSON/JSONL import, step playback, resource inspection, and native exact-replay verification.
import gymnasium as gym
import chemworld
env = gym.make("ChemWorld", task_id="reaction-to-assay", seed=0)
observation, info = env.reset(seed=0)
action = {"operation": "add_reagent", "amount_mol": 0.01}
check = env.unwrapped.validate_action(action)
if check["valid"]:
observation, reward, terminated, truncated, info = env.step(action)
env.close()Runnable notebooks for all six serious research tasks are collected in
notebooks/task_demos/. They compare public feedback across candidate
interventions and paired opaque worlds without reading hidden state or prescribing an Agent training method.
Unmeasured array values are NaN (null in JSONL). Read observed_mask or observed_keys before using an
observation field.
python -m apps.task_lab.server --port 8876- Agent Observatory: http://127.0.0.1:8876/agent/
- Student Lab: http://127.0.0.1:8876/student/
The documentation site is static; the visual interface currently runs locally. Classical agents work offline. Online model credentials are read from the local process environment and must not enter the repository or evaluation artifacts.
An optional Blender laboratory adds a mobile manipulator, explicit carrier transport, and a public-state view of Student Lab and Agent workflows. Core remains the experimental executor; the scene's educational inventory is separate. From a source checkout, launch it with:
uv run --no-sync python -m apps.blender_lab --blender /path/to/blenderSet CHEMWORLD_BLENDER_URL=http://127.0.0.1:8877 when starting Task Lab to connect the display.
For the browser explorer, run uv run --no-sync python -m apps.blender_lab --api-only --port 8878
and open http://127.0.0.1:8878/explore. It provides the exported Blender scene, 36 selectable
assets, public experiment playback, live observation history, and interactive research evidence.
Blender is not required for viewing; all web dependencies are served locally.
Physical hardware dispatch is not implemented.
The first paper releases a bounded world-building and experimental-runtime instrument. Its frozen public software and sanitized evidence are tagged v0.2.0. The release qualifies 64 reference units, 1,786 recipes and 52 generated cases, including eight new task–world identities within the supported component patterns. It does not establish arbitrary graph composition or physical-laboratory validity. Manuscript/figure assets and Work II experiments are maintained separately from this public software snapshot.
Work II studies when experimental knowledge supports scientific decisions. Two agent configurations each have 135 scheduled C2 campaigns, with 121 and 126 completed; their prediction, executable-law and decision outcomes differ. A ten-world fixed-evidence intervention (M1) found no supported material benefit from fitted-law replacement, with agent/maximizer agreement in all 40 fitted-law pairs. Information separation (M3) then showed that model laws alone improved fresh decisions relative to task-only input on new candidates in those same ten worlds: regret difference −0.13723, 95% interval [−0.15584, −0.12257]. Nearest-evidence retrieval reached the measured optimum in every world. These results establish bounded independent knowledge utility, with no demonstrated method advantage over retrieval, new-mechanism transfer, provider ranking or laboratory validity.
Current research entry points are the Work II TODO, complete story, results index and next-experiment matrix. The first-paper TODO governs its separate release. The paper guide identifies canonical manuscripts, current PDFs and build commands.
configs/current.json resolves generated evidence and separates frozen
release identity from development-head qualification and publication readiness. Current-head false
readiness flags describe ongoing development; they neither withdraw the frozen release nor qualify
new experiments. Historical reports keep their original source and scientific scope.
The website provides environment tutorials; use the repository entry points above for current research status. Pre-v0.5 algorithm results remain historical diagnostics.
| Goal | Page |
|---|---|
| Understand the research thesis | Why ChemWorld |
| Read the normative system model | System Model |
| See how causal worlds change | Causal Worlds |
| Explore Showcase Worlds | Worlds |
| Inspect Confirmatory Benchmark Tasks | Confirmatory Tasks |
| Choose an agent interaction level | Agent Tracks |
| Build an agent | Getting Started |
| Design an evaluation | Benchmark Design |
| Understand the real-world roadmap | Real-world Bridge |
Software quality gates and backend audits establish checkout integrity; they do not authorize a scientific benchmark or real-world claim. Maintainer-only release procedures are intentionally kept out of this user guide. Contributors should use the repository development and evidence guide for project maintenance workflows.