3D locomotion

StableTested and in active use. Breaking changes will be called out. Gymnasium-style tasks on physics3d; the physics is gated against MuJoCo 3.12, the task code against Gymnasium.

MuJoCo-style continuous control on noeira.physics3d — a generalized-coordinates engine with CRBA mass matrices, RNE bias forces, a Newton constraint solver and MJCF model loading. Models are defined in MJCF XML, the same format MuJoCo uses. For the DeepMind Control versions of these bodies — walker, cheetah, hopper, humanoid, quadruped, dog — see DeepMind Control.

HalfCheetah17 obs · 6 act

Hopper11 obs · 3 act

Walker2d17 obs · 6 act

Ant27 obs · 8 act

Humanoid45 obs · 17 act

Swimmer8 obs · 2 act

Environment	Obs	Act	Frame skip	Terminates early
HalfCheetah	17	6	5	no
Hopper	11	3	4	yes (unhealthy)
Walker2d	17	6	4	yes (unhealthy)
Ant	27	8	5	yes (unhealthy)
Swimmer	8	2	4	no
Humanoid	45	17	5	yes (torso z ∉ [1.0, 2.0])
HumanoidStandup	45	17	5	no
InvertedPendulum	4	1	2	yes
InvertedDoublePendulum	9	1	5	yes

InvertedPendulum4 obs · 1 act

InvertedDoublePendulum9 obs · 1 act

HumanoidStandup45 obs · 17 act

Using one

Dimensions come from the model struct:

from noeira.envs.half_cheetah import HalfCheetahModel, HalfCheetahConfig
from noeira.envs.phyics3d_batched_env import Phyics3dBatchedEnv

comptime OBS_DIM = HalfCheetahConfig.OBS_DIM   # 17
comptime ACT_DIM = HalfCheetahConfig.ACTION_DIM #  6
comptime N_ENVS = 32

comptime BatchedEnvT = Phyics3dBatchedEnv[
    HalfCheetahModel, HalfCheetahConfig, N_ENVS, TERMINATE_ON_UNHEALTHY=False
]

Swapping environments means swapping the model and config pair — network definitions that read OBS_DIM and ACT_DIM adapt automatically. See Quickstart: GPU training for a complete script.

Config or Model?

HalfCheetah and Ant expose OBS_DIM on their config; the others expose it on the model (HumanoidModel.OBS_DIM). Both are compile-time constants resolving to the same value. If one does not exist, use the other — this inconsistency is a wart, not a design.

The environments

HalfCheetah — a planar two-legged runner. Reward is forward velocity minus a control cost, with no early termination, so every episode runs the full 1000 steps. That makes returns smooth and comparable, which is why it is the default benchmark for continuous control here.

Hopper — one leg, three actuators, terminates when it falls. Deceptively hard: the healthy-state termination means a policy that has not learned to balance gets almost no data per episode.

Walker2d — two legs in a plane. Hopper’s problem with an extra limb to coordinate.

Ant — a quadruped with a free-floating torso. Twenty-seven observations, eight actuators, and enough contact points that it exercises the collision pipeline properly.

Swimmer — three links in a viscous fluid, no contacts at all. The odd one out: locomotion comes from fluid drag rather than ground reaction, so it uses a dedicated fluid force path. Notoriously hard to score well on because the reward rewards a gait most algorithms do not find.

Humanoid — the stress test. Forty-five observations, seventeen actuators, and contact-rich dynamics that expose solver problems immediately. Terminates when the torso leaves the healthy height band [1.0, 2.0].

HumanoidStandup — the same body, starting on the ground, rewarded for rising. No early termination.

InvertedPendulum / InvertedDoublePendulum — CartPole’s problem in the 3D engine. Small, fast, and useful for checking that the engine and an algorithm are wired up correctly before spending time on Humanoid.

Validation against MuJoCo

These are not approximations. The physics is checked against live MuJoCo 3.12, the version pixi installs: a family of tests steps both engines from the same state and compares trajectories, and the MJCF loader reads the same model files.

pixi run mojo run -I . tests/physics3d/test_euler_fields_vs_mujoco.mojo
pixi run mojo run -I . tests/physics3d/test_humanoid_fk_vs_mujoco.mojo
pixi run mojo run -I . tests/physics3d/test_hopper_fk_vs_mujoco.mojo

They call mj_step in-process rather than comparing against recorded fixtures. Validation has the full set and the order to debug a mismatch in.

Reward structure

All follow the Gymnasium convention:

reward = forward_reward_weight · velocity − ctrl_cost_weight · ‖action‖²  [+ healthy_reward]

Weights are compile-time constants on the config, so they are visible and adjustable without touching the environment:

comptime FORWARD_REWARD_WEIGHT = 1.25   # Humanoid
comptime CTRL_COST_WEIGHT = 0.1
comptime HEALTHY_REWARD = 5.0

One deliberate divergence: HalfCheetah’s ANGLE_PENALTY_WEIGHT is 0.0, matching reference dm_control. It was once 0.5 to suppress a “running on its head” local optimum that SAC found, but that penalty actively blocked TD-MPC2, and later optimizations made it unnecessary.

Performance notes

Frame skip — each environment step runs several physics substeps (FRAME_SKIP, 2–5 here). The physics timestep stays small for stability while the agent acts at a manageable rate.

Integrator — configurable per environment, always with the Newton solver. HalfCheetah uses Euler; the others take the environment trait’s default, RK4, and those that need it declare RK4’s extra workspace (INTEGRATOR_WS_EXTRA).

CUDA-graph capture and the blocked solver

On NVIDIA the contact solve uses a one-environment-per-block Newton kernel with cooperative shared memory. Capturing it in a CUDA graph illegal-addresses on replay, so the GPU examples set USE_ENV_CUDA_GRAPH=False while keeping USE_TRAIN_CUDA_GRAPH=True. Eager environment stepping is correct — only the per-step launch collapse is lost.

Next

• 3D manipulation — Reacher, Pusher, Sawyer
• Quickstart: GPU training — SAC on HalfCheetah end to end
• The layered stack — where physics3d sits