LiveGesture Beats Offline Models at Real-Time Avatar Motion With Zero Look-Ahead
A new framework generates full-body avatar gestures frame-by-frame from live audio, matching offline systems while running with just 250ms latency.
Takeaways
- LiveGesture is the first zero-look-ahead streaming full-body co-speech gesture model, from UNC Charlotte and UCF (paper).
- Two modules: streamable per-region VQ tokenizer plus hierarchical causal transformer with region experts and spatial-temporal fusion.
- Matches or beats offline SOTA on BEAT2: best BC 0.794, highest Diversity 13.91, FGD 4.57.
- First-token latency of 250ms, enabling live avatar conversation when paired with streaming speech models like VITA-Audio.
- Uncertainty-guided token masking plus random region masking during training reduces exposure bias from noisy streaming histories.
- Per-region codebooks and asymmetric bidirectional-encoder plus causal-decoder tokenizer are the key architectural wins.
Avatars that gesture naturally while speaking have been stuck in a frustrating tradeoff. The models that produce the best full-body motion need to see the entire audio clip before they can move a finger, which rules them out for anything interactive. A new framework called LiveGesture from researchers at UNC Charlotte and UCF tries to break that tradeoff by generating SMPL-X full-body gestures token by token from a live audio stream, with zero look-ahead into the future.
The headline claim: on the BEAT2 benchmark, LiveGesture is the only zero-look-ahead streaming method in the field, yet it hits the best Beat Constancy (0.794), the highest Diversity (13.91), and near-best Fréchet Gesture Distance (4.57 vs. 4.25 for the offline GestureLSM). It does this while producing the first motion frame roughly 250 milliseconds after the first audio token arrives.
Why streaming gestures are hard
Co-speech gesture models take audio (and sometimes text) and output 3D body motion synchronized with the speaker. The standard evaluation uses BEAT2, a dataset introduced with EMAGE that provides 60 hours of high-quality motion capture in SMPL-X format alongside speaker audio spanning a wide range of emotions. SMPL-X is a parametric body model covering face, hands, and body, so evaluating on BEAT2 tests full-body coordination, not just arm waving.
The dominant approaches, including diffusion models like DiffSHEG and autoregressive systems like EMAGE, assume access to the entire utterance. Even the fastest recent model, GestureLSM, is non-streamable: it needs the full speech segment up front. That is a dealbreaker for live avatars, VTubers, telepresence, or any conversational agent where audio arrives incrementally and the character has to move immediately.
Prior work also tends to split the body into regions and model each independently, or lump every joint into a single monolithic model. The first approach loses whole-body coordination, so arms and torso stop moving together. The second entangles high-frequency finger motion with slow torso drift into a single latent space, which quantizes badly.