Skip to main content
Back

U-JiT Beats Latent Video Models by Generating Directly in Pixel Space

A new UNet-style transformer called U-JiT generates video directly from pixels, beating latent diffusion on Kinetics-600 without needing a pretrained autoencoder.

Paper, Full Access
  • U-JiT generates video directly in pixel space, skipping the usual pretrained autoencoder stage.
  • Uses a UNet-shaped transformer with variable patch sizes to handle high-dimensional video input end-to-end.
  • Improves FVD by 27% over prior SOTA on Kinetics-600 frames-to-video at 128 resolution.
  • Lead over latent models grows as spatiotemporal compression increases up to 64x64x8.
  • On class-to-video, cuts per-frame FID by 27% vs latent and 61% vs pixel baselines.
  • Paper by Vallaeys, Verbeek and Cord; no public code release announced.

Most leading video generators compress clips with a pretrained autoencoder, then train a diffusion or flow-matching model in the resulting latent space. A paper by Théophane Vallaeys, Jakob Verbeek, and Matthieu Cord presents U-JiT, a model that generates video directly in pixel space and outperforms latent and pixel-space baselines on Kinetics-600.

Video models adopted latent generation because raw clips contain millions of pixel values. Compression can reduce that workload by a factor of 64 or more, making attention and denoising substantially cheaper. The resulting pipeline has two separately optimized components: an autoencoder trained for compression and reconstruction, followed by a generator trained on encoded clips.

That division introduces trade-offs. The autoencoder discards some visual information, and reconstruction quality does not always predict generation quality. Researchers must also choose a compression rate that balances detail, training cost, and the difficulty of modeling the latent representation.

Why pixels tend to blur

Pixel-space image models such as JiT, short for Just image Transformers, have narrowed the gap with latent image generators. Video presents a larger token problem because every frame adds another spatial grid. Keeping the sequence manageable requires larger spatiotemporal patches, which merge fine texture and motion into coarse tokens.

A fixed-width transformer struggles to recover the high-frequency information lost through that patching process. The paper links this limitation to blurred, oversmoothed frames and a gap between the principal-component spectra of generated and real clips. That spectrum measures how visual variation is distributed across the dataset.