Skip to main content
Back

CMU's COWBOY Teaches One Robot Policy to Drive, Look, and Grasp at 91%

CMU's COWBOY trains a single vision policy that drives a 32-DOF mobile robot to grasp objects across cluttered shelves, drawers, and tabletops with 91% success.

Paper, Full Access
  • CMU's COWBOY trains one vision policy for a 32-DOF mobile manipulator, zero-shot sim-to-real.
  • Hits 91% grasp success across 24 in-the-wild scenes and 100+ unseen objects.
  • Splits the problem: vectorized classical controller for reaching, RL experts for contact.
  • Whole-body controller runs 60 Hz across 4,096 parallel sim environments on one GPU.
  • Three local RL grasp experts get reused across thousands of full-scene layouts via DAgger distillation.
  • SAM3 integration adds language-conditioned target selection; paper on arXiv, no code release yet.

Mobile manipulators operating in human spaces must position the base, route an arm around furniture, aim a camera, and choose a grasp simultaneously. Carnegie Mellon University’s COWBOY coordinates those decisions with one neural policy on a 32-degree-of-freedom robot. The policy transfers directly from simulation to physical hardware without real-world fine-tuning.

Across 24 unstructured scenes containing more than 100 unique objects, the authors report a 91% grasp success rate. The evaluation includes cluttered tables, shelves, drawers, and other layouts where the policy must select an approach from fused depth-camera and lidar point clouds.

Thirty-two degrees of freedom, one coupled problem

End-to-end reinforcement learning becomes brittle when one policy must balance collision avoidance, base placement, arm motion, finger control, and active vision. A reward function covering every objective can produce local optima, while small differences between simulated and physical sensing or contact can undermine deployment.

The platform combines four actuated subsystems, with each degree of freedom representing an independently controlled axis:

Subsystem Degrees of freedom Hardware
Mobile base 3 TidyBot++
Robot arm 7 Franka arm
Dexterous hand 16 LEAP Hand
Actuated neck 6 ARX X5
Total 32 One coordinated platform

A RealSense D435 depth camera rides on the neck, allowing the robot to change its viewpoint during a task. A Unitree L2 lidar supplies additional 3D measurements of the surrounding scene.