YOLH
Imitation Learning
A robotic learning framework that enables robots to learn manipulation tasks directly from human hand demonstrations
Overview
YOLH is a robot learning project that aims to learn manipulation policies directly from human RGB video demonstrations, without requiring robot-collected data.
Traditional approaches rely on hand-hold gripper or another robot, which are expensive and difficult to scale. In contrast, human videos are abundant and easy to collect, but introduce a key challenge known as the embodiment gap—the difference between human motion and robot control.
To address this, YOLH leverages a 3D voxel-based representation to align visual observations with robot action space, enabling the model to infer executable robot actions from human demonstrations.
Fig. YOLH Pipeline. Human RGB-D demonstrations collected in simulation or the real world are processed into environment point clouds and gripper centric action labels. The final point cloud is the concatenation of the gripper point cloud, generated with action labels and URDF, and environment point cloud. The pipeline estimates hand state, removes the human embodiment, and inserts a target gripper model so the same demonstrations can be reused for different robots.
Fig. Visualization of Training Point Cloud and Simulation Evaluation.