Microsoft's AsgardBench Exposes How Badly AI Agents Fail at Visual Planning
Microsoft Research releases a benchmark that strips away navigation and rich feedback to test whether vision language models can actually replan from what they see.
Takeaways
- Microsoft Research released AsgardBench, isolating visually grounded interactive planning from navigation and low-level control.
- 108 tasks across 12 types in AI2-THOR, with object state variations that force conditional branching mid-execution.
- Claude Opus 4.5 leads at 76.5%, followed by Gemini 3 Pro (72.5%) and GPT-5.2 (71.3%); GPT-4o trails at 23.4%.
- Text-only performance collapses across models, showing prior benchmarks let agents bypass perception.
- Detailed error messages can substitute for vision in weaker models, matching image-based baselines.
- Removing hand overlay, previous-frame images, or feedback all degrade performance, exposing brittle state tracking.
Microsoft Research has released AsgardBench, a benchmark designed to answer a narrow but stubborn question about multimodal agents: can a model actually use what it sees to adapt a plan when things do not go as expected? The team argues that most existing embodied AI benchmarks let agents cheat their way to good scores using text priors or verbose error messages, and their evaluations of frontier vision language models suggest they are right.
The perception shortcut problem
Embodied benchmarks like ALFRED, ALFWorld, and EmbodiedBench mix high-level reasoning with navigation, grasping, and long detailed failure messages. In several widely used benchmarks, most notably in ALFWorld and EmbodiedBench, text only agents perform competitively or even better, and detailed failure messages often reveal why an action failed, reducing the need for genuine multimodal reasoning. That makes it hard to tell whether a model is really grounding its decisions in pixels or just pattern-matching on text.
AsgardBench targets interactive planning as the capability being evaluated and uses a controlled simulator as the evaluation setting. Agents interact with the environment through a simplified action space and receive only lightweight success or failure signals, requiring them to infer task relevant state, such as object cleanliness, container accessibility, or scene configuration, directly from images.