Skip to main content
Back

ByteDance Unveils Open-Source Agent for Human-Like GUI Interaction

TRY NOW

April 23, 2025 5 min read

Hey,

Welcome to AlphaSignal – the most read newsletter by AI developers. 


We bring you the top 1% of news, papers, models, and repos, all summarized to keep you updated on the latest in AI.

IN TODAY'S SIGNAL

Read time: 5 min 17 sec

🎖️ Top News

⚡️ Trending Signals

📌  ElevenLabs

💻  Top of YouTube

🧠  PyTorch Tip

  • Use TorchScript to cache multiple input shapes and boost inference speed with dynamic batch sizes.

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

Multimodal Agent

ByteDance unveils UI-TARS-1.5, an open-source agent with 100% scores on 14 interactive web games

⇧ 1,438 Likes

What's New

ByteDance has open-sourced UI-TARS-1.5, a multimodal agent built on the Qwen2.5-VL-7B vision-language model. It uses reinforcement learning to reason before acting, improving task execution across environments. The model reaches 100% normalized scores on 14 interactive web games, setting new benchmarks.


GUI-Based Benchmarks

The model shows high accuracy in visual grounding and desktop interaction tasks.

  • Scores 61.6 on ScreenSpotPro, compared to 43.6 (previous SOTA)

  • Achieves 42.5 on OSWorld, beating CUA (36.4) and Claude 3.7 (28)

  • Uses GUI grounding to locate screen elements with high precision

  • Operates in constrained step settings: 100 (OSWorld) and 50 (Windows Arena)

Gameplay Performance Reaches 100 Across All Tasks

UI-TARS-1.5 leads in long-horizon reasoning benchmarks using up to 1,000 interaction steps per game.

  • Achieves 100.00 across all 14 poki.com games, including Infinity-Loop and Maze: Path of Light

  • Outperforms Claude 3.7 and OpenAI CUA in all measured gameplay categories

  • Demonstrates strong scaling as interaction rounds increase

  • Evaluated using normalized scores for cross-game consistency

Web Navigation 

UI-TARS-1.5 handles structured browser tasks and information queries effectively.

  • Scores 83.8 on SimpleQA, outperforming GPT-4.5 (60)

  • Reaches 2.3 on BrowseComp, higher than GPT-4.5 (0.6)

  • Uses GUI tools to parse and interact with real-time web interfaces

  • Supports multi-step queries and structured task execution


Performance in Embodied Environments

UI-TARS-1.5 surpasses top embodied agents in Minecraft-based tasks using only visual input.

  • Beats VPT, DreamerV3, and JARVIS-VLA in Mine Blocks and Kill Mobs

  • Operates with native mouse and keyboard controls in first-person mode

  • Uses reinforcement-learned "thought-before-action" module to refine strategy

  • Shows improved task success over longer interaction windows

Architecture and Reasoning Design
UI-TARS-1.5 uses a reasoning-before-action approach based on token-level multimodal supervision.

  • Injects a “Thought” step before every UI action to improve reasoning

  • Designed to operate in real-world UI environments with minimal customization


Training Process and Data
The team trained the model using reinforcement learning and multimodal trajectory supervision.

  • Uses trajectory-weighted reinforcement learning with a self-improving critic

  • Pretrained on vision-language data and multimodal instruction datasets

  • Fine-tuned with step-level planning and memory interaction traces

  • Training includes supervised and self-play data from simulated and real UIs

Availability and Tooling for Local Use

ou can access weights, test apps, and evaluation tools through open-source repositories.

  • Open weights hosted on Hugging Face (7B parameters)

  • Desktop app includes memory, planner, and UI simulation environment

  • Tools support custom dataset integration and finetuning

  • Evaluation code includes tests for desktop, Android, and web agents

Community Feedback


Junjie Fang

"The superiority of UI-TARS-1.5 is consistent across online benchmarks, spanning computer use, browser use, and phone use."

Yiheng Xu
"I recently made a chart for a talk to track the progress of computer use agents. Just updated it with today's UI-TARS-1.5 release. The progress over the past year is mind-blowing. The climb is getting steeper. Can't stop thinking about when we'll hit that human-level singularity (72%)."

Ed

"Omg the part with the worm game. I really want to see if this thing can learn to trade. Imagine a trading bot that can be used live and also on visual information sounds amazing!"

Try now

TRENDING SIGNALS

Content Integration 

OpenAI partners with Washington Post to surface real-time reporting in ChatGPT search responses

⇧ 1,249 Likes

Text-to-Speech Model

Korean startup Nari Labs unveils Dia, a 1.6B open-source TTS model outperforming commercial leaders in expressiveness and timing

⇧ 16,858 Likes

AI Assistant

Perplexity introduces iOS Assistant; trigger Perplexity voice mode via iPhone Action button, bypassing Siri or app interface

⇧ 1,027 Likes

LLM Safety

Fiddler AI launches sub-100ms guardrails to block LLM toxicity, hallucinations, and jailbreaks in real-time with a 14-day free trial

From Our Partners

Vision Language Model

NVIDIA and UC Berkeley release DAM: Highlight any image or video region and get precise descriptions, now available on Hugging Face

⇧ 863 Like

ElevenLabs’ Scribe Turns Audio into Accurate Text

Transcribe audio with 98%+ accuracy using Scribe. Get smart diarization, audio-event tagging, and character-level timestamps, all delivered in a clean API.

Drop in, transcribe, integrate — production-ready speech-to-text for developers.

Start transcribing free today ↗️

TOP OF YOUTUBE

AI Coding Tool

The Industry Reacts to o3 and o4!

► 107,553 Views

Generative AI

Meta's AI Chief: "I'm DONE with LLMs

► 43,851 Views

Physics Based Simulation

NVIDIA’s Tech: Brutal 2,500,000 Part Simulation!

► 67,990 Views

PYTORCH TIP

Speed Up Dynamic Shape Inference with TorchScript Shape Caching

TorchScript can speed up inference even for models with dynamic input shapes using inline shape specialization. Most skip it due to dynamic batching, but torch.jit.trace now caches multiple shape patterns.


When To Use

Anytime you serve a model with varying input shapes but want low-latency inference.


Benefit

Preserves dynamic batching while enabling optimized execution via shape cache reuse.


import torch
from torch import nn

model = nn.Sequential(nn.Linear(128, 64), nn.ReLU()
                                  , nn.Linear(64, 10))
example = torch.randn(1, 128)
traced = torch.jit.trace(model, example)

# Different shape gets cached on first use
output = traced(torch.randn(32, 128))


At Alpha Signal, our mission is to build a sharp, engaged community focused on AI, machine learning, and cutting-edge language models, helping over 200,000 developers stay informed and ahead. We’re passionate about curating the best in AI, from top research and trending technical blogs to expert insights and tailored job opportunities. We keep you connected to the breakthroughs and discussions that matter, so you can stay in the loop without endless searching. We also work closely with partners who value the future of AI, including employers and advertisers who want to reach an audience as passionate about AI as we are.


Our partnerships are based on shared values of ethics, responsibility, and a commitment to building a better world through technology.Privacy is a priority at Alpha Signal. Our Privacy Policy clearly explains how we collect, store, and use your personal and non-personal information. By using our website, you accept these terms, which you can review on our website. This policy applies across all Alpha Signal pages, outlining your rights and how to contact us if you want to adjust the use of your information. We’re based in the United States. By using our site, you agree to be governed by U.S. laws.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Work with us