Skip to main content
Back

Alibaba's MeshToken Ditches 2D Rendering to Control Human Motion in 3D

Alibaba researchers replace rendered pose videos with compressed 3D mesh tokens, forcing video diffusion models to reason about geometry directly

  • MeshToken replaces rendered 2D pose videos with compressed 3D SMPL mesh tokens for video diffusion conditioning.
  • Human motion is decomposed into trajectory and canonical body pose, each encoded separately via a VQ-VAE codebook.
  • Motion tokens are injected into a frozen Wan 2.1 DiT backbone via cross-attention alongside video and text tokens.
  • The approach reduces view-dependent artifacts and trajectory-pose mismatches that plague 2D rendering pipelines.
  • Outperforms MimicMotion, RealiaDance-DiT, ControlNext, and MooreAA in qualitative benchmarks on human motion control.
  • No public code or weights released yet; project page at jingyunliang.github.io/MeshToken.

Every video diffusion model working on human motion control today shares the same fundamental assumption: to tell the model how a person should move, you render a 2D skeleton or mesh video and feed it in as a conditioning signal. It works well enough, but it quietly throws away the very thing that makes motion hard -- the full 3D geometry. A new paper from Alibaba's DAMO Academy, "Towards 3D-Aware Video Diffusion Models: Render-Free Human Motion Control with Mesh Tokenization", proposes scrapping that rendering step entirely.

Whether diffusion models are truly aware of the 3D structure underlying visual observations, rather than simply reproducing plausible 2D projections, remains an open question -- and the team investigates it through human motion control, a task that requires precise modelling of 3D human geometry, motion, camera viewpoint, and scene context. Their answer is a system called MeshToken.

Why rendered pose videos keep failing

Human-centric motion control in video generation is a critical challenge, particularly when jointly controlling camera movements and human poses. While recent video diffusion models have made significant progress, existing approaches struggle with limited motion representations and inadequate integration of camera and human motion controls.

The core issue is the rendering bottleneck. When you project a 3D SMPL body mesh onto a 2D image to create a guidance video, you lose depth, you bake in a specific camera viewpoint, and you introduce a mismatch between the trajectory of the person in the scene and the pose of their body. These view-dependent 2D guidance artifacts and trajectory-pose mismatches become especially visible during editing. The model has to guess what it cannot see.

Tokens instead of renders