Skip to main content
Back

UC Berkeley Trains Robots to Recover From Mistakes by Inoculating Their Data

A Berkeley team trains a residual RL policy to nudge human teleoperators off-course during demos, producing datasets that fix the compounding-error problem in robot learning.

Paper, Full Access
  • Berkeley's Data Inoculation adds an RL residual policy that perturbs human teleop during data collection.
  • Perturbations target the same reward used downstream, surfacing the exact mistakes a future policy would make.
  • Humans correct these induced errors in real time, producing recovery demonstrations instead of pure expert trajectories.
  • Tested on t-cube stacking, USB-C cable insertion, and keycap extraction, all precision manipulation tasks.
  • Improves or matches BC and offline RL success rates compared to standard and random data collection.
  • No public code release yet; requires a reward signal available during collection.

UC Berkeley researchers Qiyang Li, Zhiyuan Zhou, Sergey Levine, and their collaborators propose a data-collection method that deliberately exposes robots to small control errors while a human teleoperator demonstrates how to recover. Their Robotic Data Inoculation paper reports that these recovery-rich demonstrations can improve autonomous policies trained with behavioral cloning or offline reinforcement learning.

Clean demonstrations leave blind spots

Behavioral cloning trains a policy to copy expert actions from recorded demonstrations. A dataset filled with successful, efficient trajectories teaches the expected path through a task, but often omits nearby states caused by imperfect predictions, sensor noise, or minor control errors.

Those omissions create compounding error. A gripper that drifts a few centimeters may enter a state unlike anything in the training set, commonly called an out-of-distribution state. The next action becomes less reliable, the robot drifts farther, and the policy lacks examples showing how to return to a useful trajectory.

Existing methods carry practical costs. Online reinforcement learning may require extensive physical exploration. Dataset Aggregation, or DAgger, runs the current policy and asks an expert to label the states it visits, which can expose hardware to an immature controller. Informal instructions such as “vary the approach” produce inconsistent coverage and give operators little guidance about which deviations matter.

A learned residual widens the data

Data Inoculation, abbreviated DI, uses shared control during teleoperation. At each control step, the executed command combines the human action with an action offset from a residual policy: