Stanford and ByteDance's ViBES Teaches AI Avatars to Move Like Humans
Stanford researchers built a 3D avatar that jointly generates speech, facial expressions, and body motion inside one model, plus a 1,000-hour training corpus.
Takeaways
- ViBES unifies speech, facial expression, and body motion generation into one 3D conversational agent from Stanford and ByteDance.
- Uses Mixture-of-Modality-Experts with hard routing and cross-expert attention on a pretrained GLM-4-Voice backbone.
- Introduces Converse3D, a 1,000-hour audio-text-motion corpus from YouTube and existing motion datasets.
- Novel fractional RoPE aligns tokens across 12.5 fps audio and 25 fps motion streams on one timeline.
- Beats prior baselines on conversation, talking-head (LVE 10.87), and co-speech gesture (FGD 5.257) benchmarks.
- Code and data will be released at ai.stanford.edu/~juze/ViBES/, trainable on just 4 L40S GPUs.
Most conversational AI systems live inside a text box or, at best, a voice channel. A team from Stanford and ByteDance is trying to give those systems a body that actually knows when to nod, wave, or step back, and their new work reframes body language as a first-class output of the language model rather than a post-processing effect bolted onto a text response.
The project is called ViBES (Voice in Behavioral Expression and Synchrony), and it introduces a single speech-language-behavior model that jointly plans what to say, how to say it, and how the 3D avatar should move while speaking. The authors also curated a 1,000-hour audio-text-motion dataset called Converse3D to train it, since existing corpora only cover pairwise alignments like text-to-motion or audio-to-gesture.
The Frankenstein avatar problem
Today's conversational avatars are essentially three models in a trench coat: a language model produces text, a text-to-speech system renders audio, and a co-speech gesture generator produces motion clips. Prior gesture and text-to-motion systems treat behavior as a translation task, mapping a fixed utterance to motion without deciding when to move, what to do, or how to adapt across a multi-turn dialogue. The result is brittle timing, weak social grounding, and no shared conversational state across the pipeline.
Bolting a speech LLM onto a motion generator does not really fix this. There is no unified policy over timing and no shared state, so a request like "could you step back and wave?" is hard to handle because the language model does not know what the body is doing and the motion model does not know what the words mean.
One model, three experts, shared attention
ViBES is built as a Mixture-of-Modality-Experts (MoME) transformer with three modality-specific experts sharing the same backbone: a text-speech expert for language comprehension and speech generation, a facial expert for expressions, and a body expert covering upper body, lower body, and hands. Unlike sparse MoE architectures with learned routers, ViBES uses hard routing that deterministically sends each token to its modality expert, and each expert keeps its own feed-forward network, layer norms, and Q/K/V/O projections.