Skip to main content
Back

NVIDIA's PixelUMM Merges Image and Video AI Into One 8B Model

Researchers from NVIDIA and the University of Waterloo propose a unified multimodal model that reads and writes raw pixels, skipping vision encoders and VAEs entirely.

  • NVIDIA Toronto AI Lab and University of Waterloo released PixelUMM, an encoder-free unified multimodal model.
  • Operates directly on raw pixels: 16x16 image patches, 4-frame video tubes, no VAE or ViT.
  • Built on Qwen3-8B as a Mixture-of-Transformers with shared attention across text and pixels.
  • Jointly trains autoregressive text prediction and pixel-space flow matching for generation.
  • Code and 8B checkpoint available on GitHub and Hugging Face.
  • Competitive with other unified models on image and video benchmarks but trails frontier VLMs.

Researchers from NVIDIA’s Toronto AI Lab and the University of Waterloo have published PixelUMM, an 8-billion-parameter model for image and video understanding and generation. A single decoder-only Transformer maps pixel patches directly into and out of the model, removing the separate vision encoder used for understanding and the variational autoencoder commonly used for generation.

The research release includes a project page, source code, and an 8B checkpoint. Developers can inspect the architecture, reproduce its experiments, and test the trade-offs documented in the paper.

The cost of stitched-together vision

Unified multimodal models commonly combine several visual pipelines. A CLIP- or SigLIP-style encoder converts images into semantic features for understanding, while a VAE compresses pixels into a latent space for diffusion-based generation. Some systems add another tokenizer for autoregressive image synthesis. Each component brings its own parameters, interfaces, training behavior, and visual token format.

Earlier pixel-space models showed that Transformers can process images without a dedicated vision encoder. Video adds a harder representation problem: understanding benefits from sparsely sampled frames, while generation needs dense spatiotemporal regions for flow matching or diffusion. PixelUMM reconciles those formats inside one backbone.

Inside the shared backbone

PixelUMM divides images into 16-by-16-pixel patches. Its video-generation representation uses spatial patches spanning four frames, described as p16/t4. A Qwen3-8B backbone processes text, clean visual inputs, and noisy generation targets through shared self-attention layers.

The model uses a Mixture-of-Transformers design with separate feed-forward experts for understanding and generation. Shared attention allows every modality to exchange information, while the task-specific experts specialize the intermediate computation.