Stanford's DreamCan Teaches Robots to Plan With Reachable Visual Milestones
DreamCan skips dense video prediction and plans with a handful of carefully chosen keyframes, hitting 91.2% success on LIBERO-Long with a 0.1B policy.
Takeaways
- DreamCan plans robot tasks with sparse visual subgoals instead of dense video prediction, from Stanford, NUS, Harvard and Northwestern.
- Subgoals balance learned reachability and task utility, selected by dynamic programming over action-free demonstrations.
- Hits 91.2% on LIBERO-Long with a 0.1B flow-matching policy, matching 5.3B VLA baselines.
- Scales with suboptimal video data (51.6% to 68.0% on Square) by skipping bad demonstration segments.
- Anchored test-time search and VLM-written scorers enable language steering of plans.
- Code, model and dataset released: GitHub, HF models.
Video generation models encode a prior, or learned expectation, for how scenes change over time. Robot planners can use that knowledge to predict future states, but a visually plausible sequence may contain transitions that a controller cannot execute. DreamCan, developed by researchers from Stanford, NUS, Harvard, and Northwestern and led by Chongkai Gao and Fei-Fei Li, addresses that mismatch by generating a small set of visual subgoals selected for task progress and physical reachability.
DreamCan trains its high-level planner on action-free robot videos, then hands each predicted milestone to a separate goal-conditioned policy that produces robot actions. This separation allows the planner to learn from videos without control labels while reserving action-labeled trajectories for the lower-level controller.
Dense Predictions Create Brittle Plans
Existing video-based planners typically use one of three approaches, each with a distinct limitation:
- Language-only plans can leave contact points, object poses, and geometric relationships ambiguous.
- Dense video predictions require substantial computation and accumulate frame-level errors that can overconstrain the controller.
- Uniformly spaced keyframes often repeat unimportant motion, capture regressions, or place difficult transitions between adjacent goals.
DreamCan scores candidate frames using two quantities. Reachability estimates whether the robot can move from one visual state to another, while task utility measures how much a frame advances the requested task. Dynamic programming then selects the sequence with the strongest combined score.