UC Berkeley's Advantage Guidance Beats Offline RL Across 26 Sparse-Reward Tasks
A new offline RL method combines stable supervised training with contrastive diffusion guidance, beating prior methods on 26 sparse-reward tasks.
Takeaways
- UC Berkeley introduces Advantage Guidance, an offline RL method with supervised-style training and contrastive diffusion sampling. Paper
- Fixes the "advantage alignment problem" where conditional policies ignore the advantage input and collapse to behavior cloning.
- Three ingredients: action imagination, residual parameterization, and contrastive guidance between high and low advantage.
- Beats actor-critic, SL, and diffusion-guidance baselines across 26 sparse-reward tasks on OGBench and OCBench.
- Outperforms best-of-N sampling on human-like demo data while using much less test-time compute.
- Code released at github.com/zhouzypaul/advantage-guidance; recipe originated from work on Physical Intelligence's π*0.6.
Offline reinforcement learning learns from a fixed dataset, so it must improve a policy without collecting corrective experience. Actor-critic methods can optimize estimated return beyond straightforward imitation, although critic error and bootstrapped value targets often make fixed-data training sensitive to hyperparameters. Behavior cloning trains more predictably with a supervised objective, while its policy stays close to the dataset’s action distribution. In Reinforcement Learning via Advantage Guidance (paper), UC Berkeley researchers report outperforming prior offline RL methods across 26 long-horizon, sparse-reward tasks.
Zhiyuan Zhou and Sergey Levine describe Advantage Guidance, or AG, as a policy-extraction method for diffusion and flow-matching policies, which generate actions through iterative denoising or learned transport from noise. Policy fitting remains supervised; reward optimization enters through the conditioning structure and guided sampler. Zhou previously used advantage conditioning in work on Physical Intelligence’s π*0.6 vision-language-action model, which maps visual and language inputs to robot actions. AG isolates that policy-extraction recipe from the larger robotics system.
Why high-advantage requests go unheard
Return-conditioned policies such as Decision Transformer learn actions associated with requested outcomes. Advantage conditioning applies that pattern at the action level. For state s and action a, the advantage A(s,a) = Q(s,a) - V(s) estimates how much better the action is than the policy’s average action at that state. AG therefore requires a critic or another estimator that can label logged and generated actions with advantages.
- The objective matches a value. A conditional policy trained on an advantage target learns the distribution of actions associated with that target. If asked for advantage 5, it can assign lower likelihood to an action with true advantage 10 because that action does not match the requested value.