Dex3 ACT Manipulation¶

This work uses Hugging Face LeRobot's ACT policy with the Unitree G1 and Dex3 hands in Isaac Lab. ACT is an imitation-learning architecture: it predicts short chunks of robot actions from recent observations instead of selecting one action at a time.

Verified Simulation Result¶

The verified rollout uses one unified ACT checkpoint with about 34.2 million parameters. It completes a stable three-block stack in Isaac Lab using head stereo images and joint state. The validation excluded reference replay, retargeting, object snapping or pose writes, grasp assistance, and wrist-camera mounts.

This is a simulation result, not a real-robot deployment result. Earlier clips with a visually misaligned tabletop and staged checkpoint switching are superseded by this unified-policy rollout.

Local Project State¶

The checkout has local changes in unitree_lerobot/eval_robot/eval_g1.py, the LeRobot submodule, camera configuration, and scripts. Preserve those changes before updating or cleaning the repository.

Data Path¶

demonstration episodes
  -> LeRobot dataset
  -> ACT training
  -> checkpoint
  -> Isaac Lab policy rollout
  -> physical G1 evaluation only after hardware and safety gates

For new demonstrations, the Quest path can supply hand and arm motion while Isaac Lab supplies synchronized camera and joint observations. Dataset schema, camera placement, action dimensions, normalization statistics, and timing must match between training and inference.

References¶

• Unitree LeRobot
• Hugging Face LeRobot
• ACT policy documentation