Stanford's BrickBench Forces AI Agents to Build Real LEGO Models
Stanford, Max Planck, and Inria release a 300 prompt benchmark that asks coding agents to design buildable LEGO sets, exposing a sharp gap with human designers.
Takeaways
- Stanford, Max Planck, and Inria released BrickBench, 300 prompts for agentic LEGO design.
- Three settings: Model (under 400 parts), Set (400 to 4000), and Alt-Build (fixed inventory of set 10698).
- Scores physical validity, semantic alignment via VQA, and design quality via VLM ELO ratings.
- With BrickAgent tools, top agent built valid assemblies on all 300 tasks and hit 95 percent semantic accuracy.
- Removing the environment dropped validity to 40 percent, isolating tool use as the decisive factor.
- Humans still beat agent designs in 323 of 360 pairwise comparisons; code on PyPI.
Automating a LEGO model forces an agent to solve catalog search, geometry, structural stability, and visual interpretation in one workflow. Researchers at Stanford, the Max Planck Institute for Intelligent Systems, and Inria have released BrickBench, a benchmark that grades frontier coding agents on those combined demands. Its 300 text prompts test whether agents can turn a description into a stable, recognizable brick assembly.
Each submission receives scores across three dimensions:
- Physical validity: The simulator checks legal connections, part collisions, and whether the structure remains standing under gravity.
- Semantic alignment: Yes-or-no visual questions derived from the prompt check whether the finished model includes the requested features.
- Design quality: A vision-language model compares pairs of builds and produces Elo ratings. The researchers calibrated those judgments against human raters.
The research paper uses brick design to test local and global reasoning in the same task. An agent must align individual studs and connectors while preserving the proportions, structure, and recognizable features of the requested object.
Three parts bins, three constraints
The benchmark divides its 300 prompts into three settings based on the inventory available to the agent:
| Setting | Available inventory | Constraint |
|---|