Meta's Stiffness Copilot Cuts Robot Contact Force in Half During Delicate Tasks
Meta Reality Labs built a vision policy that watches a wrist camera and reshapes a robot arm's stiffness on the fly during teleoperation.
Takeaways
- Meta Reality Labs' Stiffness Copilot predicts a full 3x3 robot stiffness matrix from wrist-camera images at 90 Hz.
- Labels come from simulated contact forces; a frozen DINOv3 backbone plus small MLP transfers zero-shot to real cameras.
- In an 18-person user study, it matched low-stiffness safety and high-stiffness speed across three contact-rich tasks.
- Vase wiping success rate jumped from 24% (high stiffness) to 87% with the copilot; peg force halved.
- Users rated the copilot significantly more controllable, predictable, and trustworthy with lower NASA-TLX workload.
- Limitations: translational stiffness only, quasi-static contact, no code released, and no unseen-task generalization tested.
Teleoperating a robot arm through a contact-rich task like wiping a vase or seating a peg into a hole is a balancing act. Push too hard and you trigger a protective stop or damage the object. Move too gingerly and the arm feels like it is dragging through mud. A team from Meta Reality Labs Research, with collaborators at the University of Wisconsin-Madison and Purdue, has released Stiffness Copilot, a vision-based policy that watches a wrist-mounted camera and quietly re-tunes the arm's stiffness matrix while a human drives the end effector.
The paper, accepted at IROS 2026, frames the trade-off cleanly. Constant low stiffness improves safety but reduces responsiveness. Constant high stiffness improves efficiency but increases impact risk. Stiffness Copilot tries to get both at once by predicting a full 3x3 stiffness matrix (not just a scalar) from a single RGB frame, at 90 Hz.
Why constant impedance keeps breaking things
Most consumer-grade teleoperation rigs use lightweight input devices like VR controllers or a SpaceMouse, which typically provide limited or no force feedback. That makes contact-rich manipulation difficult because operators cannot sense how the robot is loading the environment. When the operator's hand slips a few millimeters off the arc of a cabinet door, the arm keeps pushing against the hinge and the interaction force spikes.
Prior fixes fall into a few uncomfortable buckets. Some interfaces ask the operator to manually shape a stiffness ellipsoid with a tablet or foot pedal, but tuning a full matrix by hand is cognitively demanding and often requires domain expertise. Tele-impedance systems read the operator's own arm stiffness with EMG electrodes, which works but requires strapping physiological sensors to a human. Rule-based schemes flip between a couple of discrete modes based on measured force, and tend to be brittle across tasks.