Skip to main content
Back

ViewMask-1-to-3 Beats Stable Diffusion at 3D Views 4x Faster

A discrete diffusion model treats multi-view image generation as masked token prediction, outperforming continuous diffusion baselines on standard 3D benchmarks.

Paper, Full Access
  • ViewMask-1-to-3 reframes multi-view generation as masked discrete token prediction instead of continuous diffusion.
  • Uses MAGVIT-v2 tokenizer with 2^18 vocabulary; simple random masking yields cross-view consistency without 3D priors.
  • Ranks first on average across PSNR, SSIM, LPIPS, CD, IoU on GSO and 3D-FUTURE benchmarks.
  • Beats continuous diffusion baselines by 10.6% IoU on 3D-FUTURE; 1.28s per view at 8B parameters.
  • Same checkpoint handles T2I (0.72 GenEval), captioning, inpainting, and training-free 2D turn-style rotation.
  • Limitations: 256x256 resolution, fixed 30-degree elevation training data, lighting robustness still weak.

Generating multiple consistent views of an object from a single image has long been the territory of continuous diffusion models, the same family that powers Stable Diffusion and its descendants. A team from Northwestern Polytechnical University, TeleAI, Huawei and the University of Hong Kong just pushed back on that assumption. Their new system, ViewMask-1-to-3, treats each viewpoint as a sequence of discrete visual tokens and generates them through masked token prediction, the same trick behind BERT and recent diffusion language models like LLaDA.

The result is a single model that beats a stack of specialized continuous-diffusion baselines on multi-view image quality and geometry, while also handling text-to-image, image captioning, and inpainting from the same weights.

Why continuous diffusion has owned this task

Multi-view generation sits at the intersection of 2D image synthesis and 3D reasoning. Given one photo of a chair, you want three or four new views that agree about leg thickness, seat curvature, and texture. Existing approaches fall into three camps: 3D representation methods like TRELLIS or DreamGaussian that build an explicit 3D structure and render views from it, camera-conditioned diffusion models like Zero-1-to-3 that fine-tune a 2D diffusion model on camera poses, and image-editing methods like MV-Adapter that repurpose text-to-image models for view synthesis. All of them operate in continuous latent space and typically need either 3D priors, epipolar geometry constraints, or specialized attention modules to keep views consistent.

Comparison of four multi-view generation approaches

The problem the authors flag is that discrete diffusion has quietly become a serious competitor in language and vision-language modeling, but nobody had tried it on a geometrically demanding task like this. Masked diffusion models offer faster inference than autoregressive models, native fusion of text and visual tokens, and natural alignment with large language models. If it works here, you get a unified token-space model instead of a zoo of task-specific pipelines.