Whole-Body Dexterity

Anonymous Authors

Abstract

Current humanoid dexterous manipulation remains largely confined to end-effectors and simple grasps, falling short of humans’ ability to engage the entire body by bracing and sliding bulky objects against the arms and torso. We introduce Whole-Body Dexterity (WBD), a general framework that uses the robot's body and limbs as contact surfaces to grasp and manipulate objects. Our sim-to-real pipeline generates a cache of feasible whole-body grasps, trains reinforcement-learning experts on an any-to-any grasp objective, and distills them into a single deployable proprioception-only policy. The distilled policy transfers zero-shot to a real Unitree G1, repositioning a basketball and cardboard box, completing 90% of trials, compared with 18% for direct retargeting and 4% for a generalist whole-body controller.

90%WBD completed trials
45/50 · distilled controller
18%GMR completed trials
9/50 · direct motion retargeting
4%SONIC completed trials
2/50 · generalist whole-body controller

Teleoperation setup

All trials are run by a single skilled teleoperator wearing an Xsens motion-capture suit, which streams SMPL body motion that serves as a motion command. The operator commands the upper body, while a separate controller handles the lower body.

Long-horizon shelf-to-table task

The shelf task combines several dexterous skills into one sequence: pick a box from a shelf, stow it under one arm, carry it to a table, and place it.

Shelf-to-table task
ControllerPick ↑Stow ↑Carry ↑Un-stow ↑Place ↑
SONIC10/100/100/100/100/10
GMR10/103/102/101/101/10
WBD (ours)10/109/109/109/107/10

On-body repositioning

The object moves from a stable two-arm front grasp to a target region on the body and back to a stable two-arm front grasp.

Basketball (size 3)
TaskControllerReached ↑Round trip ↑Region (s) ↓Total (s) ↓
Shoulder tapSONIC2/100/1015.2—
GMR3/101/1026.030.1
WBD (ours)10/109/107.717.3
Underarm stowSONIC1/100/1028.0—
GMR4/102/1031.549.8
WBD (ours)10/109/1010.626.2
Cardboard box (15 cm cube)
TaskControllerReached ↑Round trip ↑Region (s) ↓Total (s) ↓
Shoulder tapSONIC2/102/1015.924.9
GMR6/104/108.08.5
WBD (ours)10/1010/104.16.3
Underarm stowSONIC2/100/1026.1—
GMR2/101/109.317.8
WBD (ours)10/1010/108.613.0