Skip to main content
Back

UC Berkeley's OmniTaskonomy Cuts Visual Training Data Tenfold With Image Generation

A controlled study across 19 generation and 25 understanding tasks shows that pretraining on image generation can reduce understanding data needs by up to 10x.

  • OmniTaskonomy maps transfer between 19 image generation tasks and 25 understanding capabilities in unified models.
  • Pretraining on 100k generation examples cuts labeled understanding data needs by up to 10x.
  • Recipe matters: I2I pretraining then I2T fine-tuning works, joint mixing from scratch does not.
  • Depth and surface normals boost 3D reasoning by 3.4 to 3.8 points; jigsaw lifts 2D ordering 6.8 points.
  • Gradient alignment in early pre-attention RMSNorm layers predicts transfer gains (r=0.795 across capabilities).
  • Code, models, and paper are public.

Researchers from UC Berkeley, Duke, Carnegie Mellon, and collaborating institutions report that image-generation pretraining can improve a unified model’s visual question answering when training happens in stages and the generation task aligns with the target skill. Their strongest schedule uses image-to-image pretraining to update shared model weights before image-to-text fine-tuning.

The OmniTaskonomy preprint tests which generation tasks transfer to specific understanding capabilities and examines the mechanism behind those gains. The authors also provide a project site, source code, and a model collection.

One Problem, Two Output Modes

Unified vision systems such as BAGEL, Chameleon, and Transfusion process images and text within one model. Image-to-image tasks, abbreviated I2I, produce an image from an image input. Image-to-text tasks, or I2T, answer questions about an image in text. Prior studies often combined these objectives during training or isolated them completely, making the source of any transfer difficult to identify.

To isolate output modality, the authors created paired tasks with the same input and underlying problem. Only the required output changes. In the Jigsaw task, for example, the I2I model produces a reassembled image, while the I2T model returns a permutation such as ('reorder', [3, 2, 1, 0]). The paired benchmark applies the same design to Zoom-In tasks.

Stage Generation First

The authors evaluated six schedules on a Mixture-of-Transformers model, an architecture with modality-specific processing paths and parameters shared across image and text. The relevant schedule groups produced distinct results:

Training schedule Reported behavior
I2I pretraining with shared weights updated, followed by I2T fine-tuning Transfer improved consistently as the I2I dataset grew.
I2I pretraining with shared weights frozen