U-JiT Beats Latent Video Models by Generating Directly in Pixel Space
A new UNet-style transformer called U-JiT generates video directly from pixels, beating latent diffusion on Kinetics-600 without needing a pretrained autoencoder.
Takeaways
- U-JiT generates video directly in pixel space, skipping the usual pretrained autoencoder stage.
- Uses a UNet-shaped transformer with variable patch sizes to handle high-dimensional video input end-to-end.
- Improves FVD by 27% over prior SOTA on Kinetics-600 frames-to-video at 128 resolution.
- Lead over latent models grows as spatiotemporal compression increases up to 64x64x8.
- On class-to-video, cuts per-frame FID by 27% vs latent and 61% vs pixel baselines.
- Paper by Vallaeys, Verbeek and Cord; no public code release announced.
Most leading video generators compress clips with a pretrained autoencoder, then train a diffusion or flow-matching model in the resulting latent space. A paper by Théophane Vallaeys, Jakob Verbeek, and Matthieu Cord presents U-JiT, a model that generates video directly in pixel space and outperforms latent and pixel-space baselines on Kinetics-600.
Video models adopted latent generation because raw clips contain millions of pixel values. Compression can reduce that workload by a factor of 64 or more, making attention and denoising substantially cheaper. The resulting pipeline has two separately optimized components: an autoencoder trained for compression and reconstruction, followed by a generator trained on encoded clips.
That division introduces trade-offs. The autoencoder discards some visual information, and reconstruction quality does not always predict generation quality. Researchers must also choose a compression rate that balances detail, training cost, and the difficulty of modeling the latent representation.
Why pixels tend to blur
Pixel-space image models such as JiT, short for Just image Transformers, have narrowed the gap with latent image generators. Video presents a larger token problem because every frame adds another spatial grid. Keeping the sequence manageable requires larger spatiotemporal patches, which merge fine texture and motion into coarse tokens.
A fixed-width transformer struggles to recover the high-frequency information lost through that patching process. The paper links this limitation to blurred, oversmoothed frames and a gap between the principal-component spectra of generated and real clips. That spectrum measures how visual variation is distributed across the dataset.