feat(lerobot): add lerobot eval support by kainotkai · Pull Request #36 · GPUOpen-LibrariesAndSDKs/Schola

kainotkai · 2026-08-07T18:54:11Z

Summary

Related issues

Type of change

New feature or enhancement

Changes

Add support for LeRobot

Testing

Python (`Resources/python`, `Test`)

Installed test dependencies: pip install --group test -e "./Resources/python[all]"
Ran: python -m pytest Test --import-mode=importlib -n 0
Not applicable

Unreal C++ (`Source`)

Built / recompiled the project after Unreal Engine changes
Ran relevant automation tests (editor or pytest on Windows; see CONTRIBUTING.md)
Not applicable

Checklist

Code follows project style (Unreal coding standard for C++; Black for Python)
Comments / docstrings added or updated where behavior is non-obvious
README or Sphinx docs updated if user-facing behavior changed
No unrelated changes included in this PR
Appropriate license headers added to new files

amd-alexcann · 2026-08-10T21:20:01Z

+
+
+class LeRobotScholaVectorEnv(gym.vector.VectorWrapper):
+    """Adapt Schola's native vector environment to LeRobot's rollout contract."""


Docs and typing are mismatched here, docs are more restrictive than typing allows (docs say Schola Env, init says any VectorEnv).

Updated the documentation. Thanks

amd-alexcann · 2026-08-10T21:22:25Z

+        self._validate_render_camera()
+
+    def _validate_render_camera(self) -> None:
+        pixels_space = self.single_observation_space.spaces.get("pixels")


Is it possible to make this something exposed on the CLI rather than hardcoded?

e.g. --camera-obs-key "pixels", then use that to get the value (or map to pixels if pixels is special in LeRobot)

Currently, we don't support the rendering modes from gymnasium etc. and instead handle it ourselves by passing NullRHI etc. I would recommend continuing that here as a permanent fix should go in the core Schola modules.

If we absolutely need a render camera to do anything then let's revisit

pixels is LeRobot's harcoded image-observation key (not Schola's observation key).ObservationAdapter uses observations.cameras to map Schola image keys to LeRobot's pixels/<camera> structure. line 84 for similar example

_validate_render_camera() just verifies that the already-converted LeRobot observation space is what LeRobot expects (pixels/<camera>)

render_camera selects one of the cameras in the observation space. The render method just returns the last frame to assemble MP4 videos (ex. to see how a robot is doing). It doesn't initialize a separate Unreal render.

amd-alexcann · 2026-08-10T21:25:22Z

+    passthrough: dict[str, str] = field(default_factory=dict)
+    """LeRobot output name to an unchanged Schola observation key."""
+
+    ignore: list[str] = field(default_factory=list)


When would we intentionally ignore observations from Schola? (instead of just omitting them at the Unreal level)

Removed ignore. I agree that the use case is way too narrow and impractical

amd-alexcann · 2026-08-10T21:27:15Z

+
+SINGLE_IMAGE_NDIM = 3
+BATCHED_IMAGE_NDIM = 4
+CHW_CHANNEL_AXIS = -3


Terminology flips between AXIS and DIM here. Prefer DIM for a single dimension, and DIMS or NDIMS plural when referring to multiple.

Updated to use NDIMS instead

amd-alexcann · 2026-08-10T21:29:56Z

+            "within [0, 1]"
+        )
+
+    # Prefer channel-last when both edge dimensions look like channel counts.


Schola is on CHW (there is a bug in the camera that means it could look otherwise but my open PR resolves the issue by standardizing on this)

updated to use CHW

amd-alexcann · 2026-08-10T21:31:58Z

+BATCH_AXIS = 0
+
+
+def _convert_image_space(space: gym.Space, name: str) -> tuple[Box, str]:


I suspect this function and some of the other helpers are reinventing the wheel, please check existing libraries (gymnasium, lerobot for similar pre-existing helpers. I know SB3 has a helper along these lines but don't bring it into this library just for the helper)

Updated and used some Gym helpers

amd-alexcann · 2026-08-10T21:44:43Z

+                "Schola manages vectorization through GymVectorEnv; "
+                "LeRobot async environment wrapping is not supported."
+            )
+        if self.observations.is_empty():


What specifically do we absolutely need, that can't be inferred, potentially by doing the same trick as rllib where we spin up an environment to get the metadata?

Unreal Engine might expose joint_positions, joint_velocities, and gripper. If the policy was trained on one combined observation.state vector we wouldn't be able to know how to combine them and in what order.

observations identifies the grouping and order.

Ex: I got policy online and i need to ensure I pass it as joint_positions, joint_velocities, and gripper in that order. However, without observations, we don't know which order to specify how the state must be assembled.

amd-alexcann · 2026-08-10T21:47:20Z

+            verbosity=self.verbosity,
+        )
+
+        if schola_env.num_envs != n_envs:


Can we just warn and ignore the LeRobot num_envs? If we can't it might be best to implement this as a GymEnvironment and leave the vectorization up to lerobot.

Let me know what options we can't control, and we can discuss

Added a warning. We can just use the actual env.num_envs exposed by Schola

amd-alexcann · 2026-08-10T21:50:00Z

+            raise
+
+        try:
+            if bool(self.features) != bool(self.features_map):


Why do we need two separate maps to handle this? What is the ground truth for the keys in these maps?

features is a PolicyFeature that specifies the type (state, visual, action, etc.) and the shape.

features_map renames the feature to the name LeRobot expects.

While we could find a way to combine them with one structure, it seems like the norm is to pass them both as two different maps. Example here.

kainotkai added 5 commits August 7, 2026 14:47

feat(lerobot): add eval support

a661d9d

refactor observation_map

45175d8

linting

9bc36e0

documentation

42a40e5

fix liscence headers

7df563d

kainotkai requested a review from amd-alexcann August 10, 2026 18:36

kainotkai marked this pull request as ready for review August 10, 2026 18:37

kainotkai changed the title ~~feat(lerobot): add lerobot support~~ feat(lerobot): add lerobot eval support Aug 10, 2026

fix lerobot version

61976c5

amd-alexcann reviewed Aug 10, 2026

View reviewed changes

implement code review feedback

7922e1d



		class LeRobotScholaVectorEnv(gym.vector.VectorWrapper):
		"""Adapt Schola's native vector environment to LeRobot's rollout contract."""

		BATCH_AXIS = 0


		def _convert_image_space(space: gym.Space, name: str) -> tuple[Box, str]:

Uh oh!

Conversation

kainotkai commented Aug 7, 2026

Summary

Related issues

Type of change

Changes

Testing

Python (Resources/python, Test)

Unreal C++ (Source)

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kainotkai Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Python (`Resources/python`, `Test`)

Unreal C++ (`Source`)

kainotkai Aug 11, 2026 •

edited

Loading