Reproduction environment for the released PUMA checkpoint

[Huni0318]

Hi, thank you for releasing PUMA and DOMINO.

I evaluated the released steps_100000_pytorch_model.pt checkpoint, but obtained a substantially lower result:

• Paper: 17.20% SR
• Bundled log/log.csv: 19.23% SR
• My evaluation: 10.63% SR

The checkpoint SHA256 matches the Hugging Face release:

45af1158227071cf1e9d2bcb5ad6c9b9945987e0ee3c8384ff5c449963737fdc

For adjust_bottle, 99 of the 100 accepted seeds were shared between the bundled evaluation and my evaluation. On these common seeds, the bundled result had 70 successes, while my evaluation had 31. Therefore, the gap does not appear to come only from different accepted seed sets.
My evaluation used demo_clean_dynamic, Level 1, coefficient 0.1, 100 episodes, and the checkpoint's original 4-frame optical-flow history and 16-action execution horizon.
Could you please clarify the following environment details?
What exact DOMINO git commit and task configuration were used for the bundled checkpoint evaluation?
Is the bundled result's aloha-agilex_clean_level1 configuration identical to the public demo_clean_dynamic configuration?
What CUDA, PyTorch, SAPIEN, MPLib, and CuRobo versions were used for the simulator and PUMA policy server?
Were the policy server and simulator run on the same RTX 4090, on separate GPUs, or on separate machines?
Was the RTX 4090 mentioned in the paper used only for latency measurement or also for the full DOMINO evaluation?
If possible, could you also provide the original aloha-agilex_clean_level1.yml file or an environment specification used for the released checkpoint evaluation?
Thank you for your help.

[FangHeng]

Hi, thanks for the report and for verifying the checkpoint hash. Let me address your questions one by one.

On the checkpoint and the reported numbers. The released checkpoint is the same model framework as in the paper. For the open-source release we reorganized and cleaned up the code implementation and retrained the model with it, and the bundled 19.23% comes from re-evaluating that retrained checkpoint. So the gap between 17.20% and 19.23% is simply the variation from a fresh training run on the reorganized codebase, not simulator instability. As for the weights themselves, we are confident they are fine. We have reproduced the bundled results on multiple machines, community users we have been in contact with also reproduced them, and our recently released Ascend NPU inference path serves the exact same checkpoint with essentially identical numbers. The specific successful episodes can vary slightly across machines, but in all our tests the impact on overall SR is small, usually under 2%. A drop from ~19% to ~10% therefore points to the policy server or simulation environment rather than the weights.

Git commit and task configuration. Any commit after 644a2c297f39af08db103ac3d51cdd3a1ca47dc4 is fine, none of the changes since then are breaking. aloha-agilex_clean_level1 is identical to the public demo_clean_dynamic. During internal testing we copied the config per embodiment and dynamic level purely for bookkeeping, and merged them into a single demo entry on release. The underlying config is exactly the same, so there is no separate yml to provide.

Policy server / simulator placement. This does not matter. The two sides communicate over websockets, so same-machine or cross-machine makes no difference to inference results. For NVIDIA inference we ran both on one machine (4090 or 3090), for real-robot and Ascend we used cross-machine remote inference, with no performance difference.

About the RTX 4090. The 4090 in the paper only standardizes the hardware for latency measurement. DOMINO is not bound to any specific GPU. Since SAPIEN relies on ray tracing we recommend RTX series cards, as rendering issues have been reported on A/H series GPUs (see RoboTwin issue #83 and SAPIEN issue #219). We have run the simulation on 4090, 3090 and L20, all stable.

Where I would look for the gap. Three places, roughly in order of likelihood:

1. Policy setup. In our experience an SR around 10% on this setting looks very much like the optical-flow input not being properly enabled. Please double check that optical flow is fully turned on during eval and that the flow inputs are actually computed and fed to the model. Also note that for some specific libraries, transformers in particular, different versions lead to different inference results, which is exactly why we pin the necessary versions in the requirements. Our policy environment is built following policy/PUMA/README.md, with CUDA 12.4 following the StarVLA setup.

2. Simulation setup. Our simulation environment follows the RoboTwin setup with CUDA 12.1 and is built from the repo's install scripts. On cuRobo specifically, we have run checkouts ranging from v0.7.6 up to recent main across our machines and never observed a performance difference of this magnitude, so cuRobo is unlikely to be the cause. If your simulation environment has an issue it would not be limited to DOMINO, so a quick sanity check is to reproduce a standard RoboTwin baseline result with the same environment.

3. Hardware / rendering. Since we currently know very little about your hardware and runtime environment, we cannot rule out a low-level rendering problem on your setup. You can inspect the collected observations and the videos saved during eval for rendering artifacts, and verify that vanilla SAPIEN and RoboTwin run correctly on your machine.

That said, the information available so far is still quite limited for pinpointing the cause. If you could share more about your setup, such as your hardware and driver details, the environments of both sides, and some eval artifacts, we would be happy to help narrow down where the problem is.