Stanford's RePR Trains Image Encoders 8x Cheaper Than DINOv2
A new self-supervised tokenizer called RePR scales predictably with compute, hits FID-50K of 2.18 on ImageNet, and beats raw diffusion.
Takeaways
- RePR is a new self-supervised tokenizer training recipe for generative image models from Stanford.
- It estimates pointwise mutual information between masked views using a jointly trained generative model instead of contrastive negatives.
- Hits FID-50K of 2.18 on ImageNet with JiT-L, improving over 2.36 baseline.
- Scaling-law extrapolation suggests 7x less compute than DINOv2 and 8x less than WebSSL.
- Avoids feature suppression that collapses DINO and LeJEPA on watermarked data.
- Code available at github.com/AlanBaade/repr.
Most latent image generators first use an encoder, often called an image tokenizer, to compress pixels into a smaller grid of latent vectors. A diffusion model then learns to generate those vectors, reducing training and inference costs. Tokenizer quality has improved less predictably with added compute than downstream diffusion models, leaving the encoder as a bottleneck. Stanford researchers address that constraint in the RePR paper, Generative Representation Learning for Scalable Latent Tokenization, which treats representation training as its own scaling axis.
Alan Baade, Kyle Sargent, Vincent Sitzmann, Eric Ryan Chan, Justin Johnson, Li Fei-Fei, and Gordon Wetzstein report an FID-50K of 2.18 on ImageNet class-conditional generation with a large JiT diffusion backbone. Their scaling-law analysis also projects that RePR can match DINOv2 and WebSSL representation quality with one-seventh to one-eighth as much tokenizer-training compute.
Measured gains, modeled savings
| Claim | Reported result | Evidence |
|---|---|---|
| Generation quality | 2.18 FID-50K on ImageNet with JiT-L. FID compares 50,000 generated samples with real images; lower scores indicate closer distributional similarity. | Reported experiment |
| Tokenizer efficiency | One-seventh to one-eighth as much tokenizer-training compute as DINOv2 and WebSSL at matched representation quality. | Scaling-law extrapolation |
| Compute allocation | Roughly 20% to 35% of total training FLOPs assigned to the tokenizer in the fitted compute-optimal regime. | Scaling-law analysis |
| PMI estimator cost | About 1.3% of total training FLOPs using roughly 32 integration steps. | Training measurement |