Skip to main content
Back

SUSTech's GameUIAgent Turns Text Into Editable Figma Game UIs

A neuro-symbolic pipeline turns text prompts into editable Figma game cards, exposing a quality ceiling that limits how much iterative refinement can help.

  • GameUIAgent converts text prompts into editable Figma game UIs via a Design Spec JSON intermediate representation.
  • Six-stage pipeline combines LLM generation, deterministic post-processing, and a VLM-driven Reflection Controller loop.
  • DeepSeek V3 hit 98% JSON validity and 8.0/10 quality; Gemini and GPT-4o-mini trailed significantly.
  • Quality Ceiling Effect: iterative refinement gains correlate at r=-0.96 with initial score, so evaluators become the bottleneck.
  • Partial renderer upgrades can invert VLM rewards by exposing layout defects that flat colors previously hid.
  • Code and 110-case eval suite released on GitHub; RPG gacha only for now.

Game UIs are notoriously repetitive to produce. Every character card, item thumbnail, and skill panel typically ships in five rarity tiers, each with its own borders, glows, and badges. A team from SUSTech, Tsinghua, and the Zhongguancun Academy just published GameUIAgent, an LLM-powered pipeline that converts a plain text description into a fully editable Figma design, and along the way it surfaces two empirical rules that anyone building visual generation agents should probably internalize.

Why raster generators keep failing at game UI

Most generative UI tools either target web layouts or spit out static PNGs that a designer cannot edit downstream. GameUIAgent takes a different route. Figma-integrated approaches like Figma Make, GUIDE, and CoGen operate single-shot without quality verification and lack game-domain rules such as rarity hierarchies. Gacha-style games encode a strict visual hierarchy (N, R, SR, SSR, UR) where higher rarity means more badges, richer gradients, and ornate frames, and no off-the-shelf model respects that structure.

Five rarity tiers N through UR with progressively richer card designs

A JSON schema that sits between the LLM and Figma

The core trick is a Design Spec JSON intermediate representation. The schema defines a recursive node tree in which each node specifies geometry, visual style, text content, and hierarchical relationships, with four node types: FRAME, RECTANGLE, ELLIPSE, and TEXT. A token budget of 6,000 per call accommodates complex UR-tier designs. Because the schema uses natural naming that aligns with LLM pre-training data, it plays nicely with any OpenAI-compatible endpoint, and because it maps to universal vector primitives, the same JSON can render to Sketch or Adobe XD.