Skip to main content
Back

Microsoft's OmniParser V2 Turns Any LLM Into a Computer-Use Agent

Microsoft's screen parsing model turns any vision-capable LLM into a computer use agent, cutting latency 60% and hitting 39.6% on ScreenSpot Pro.

  • Microsoft released OmniParser V2, an open source screen parser that turns any vision LLM into a computer use agent.
  • Hits 39.6% on ScreenSpot Pro with GPT-4o, versus 0.8% for raw GPT-4o on the same benchmark.
  • 60% latency reduction over V1: 0.6s per frame on A100, 0.8s on RTX 4090.
  • Ships with OmniTool, a dockerized Windows 11 VM sandbox for end-to-end agent experimentation.
  • Works out of the box with GPT-4o, o1, o3-mini, DeepSeek R1, Qwen 2.5VL and Anthropic Sonnet.
  • Split license: icon caption model is MIT, icon detection model is AGPL, weights free to download.

Microsoft has released OmniParser V2, an open source screen parsing model that converts UI screenshots into structured lists of clickable regions and icon descriptions. It is the glue layer that lets a general purpose language model like GPT-4o or Claude Sonnet see and click things on a desktop, without needing a specialized GUI-native model.

Drop OmniParser in front of any vision LLM and you get a computer-use agent. It tokenizes screenshots from pixel space into structured elements, which the downstream LLM can then reason over and act on. Microsoft frames the release as a way to turn any LLM into a computer-use agent.

What actually changed under the hood

Two models ship in the checkpoint: a finetuned YOLOv8 for detecting interactive regions, and a finetuned Florence-2 base model that generates a functional caption for each icon. V2 is trained on a larger and cleaner grounding and captioning dataset than V1, and Microsoft shrunk the input image size fed to the caption model to cut inference cost.

The numbers on the release are concrete:

  • 60% latency reduction versus V1, averaging 0.6s per frame on an A100 and 0.8s on a single RTX 4090
  • 39.6 average accuracy on the ScreenSpot Pro grounding benchmark when paired with GPT-4o, up from GPT-4o's baseline score of 0.8