Sakana AI's DiffusionBlocks Trains Giant Neural Networks One Block at a Time
Sakana AI's DiffusionBlocks trains neural networks one block at a time, cutting GPU memory proportionally while matching end-to-end performance across five architectures
Takeaways
- Memory cut proportionally: DiffusionBlocks trains one block at a time, reducing GPU memory by a factor of B (number of blocks) with no performance loss.
- Diffusion theory as the key: Residual connections map to ODE steps in a diffusion process, giving each block a principled, independent training objective.
- Five architectures validated: ViT, DiT, masked diffusion, autoregressive, and recurrent-depth transformers all match or beat end-to-end baselines.
- 4× memory cut on LLMs: Autoregressive transformers maintain generation quality with 4× less memory; masked diffusion beats MD4 with 3× less memory.
- Looped Transformer bonus: Replaces expensive K-iteration backprop-through-time with a single forward pass during training.
- Open source: Code available at github.com/SakanaAI/DiffusionBlocks; paper at arxiv.org/abs/2506.14202.
Training a large neural network has always meant one thing: hold the whole thing in memory at once. Every layer, every activation, every gradient -- all of it resident during the backward pass. DiffusionBlocks, a new framework from Sakana AI and the University of Tokyo presented at ICLR 2026, breaks that assumption. It trains networks one block at a time, and the memory cost scales down proportionally.
Standard neural network training optimizes all parameters jointly, so the memory it requires grows with model size. Block-wise training instead trains each block of the network independently, using memory for just one block at a time. The catch has always been: how do you give each block a meaningful training signal without the rest of the network? That is exactly what DiffusionBlocks solves.
Why memory is the real wall
End-to-end backpropagation requires storing activations throughout all layers, creating memory bottlenecks that limit model scalability. As Transformers are scaled primarily by adding more layers, depth directly drives the growing memory cost. Today's frontier models typically have hundreds of billions of parameters or more and require thousands of GPUs to train, and only a small number of organizations have the resources to develop them.
One existing technique, activation checkpointing, reduces activation memory by recomputing activations on demand. However, it does not reduce memory for parameters, gradients, or optimizer state. Block-wise training is a fundamentally different lever -- instead of optimizing how you store things, you simply never load most of the network in the first place.
Existing block-wise training methods offer means to alleviate this problem, but they rely on ad-hoc local objectives and remain largely unexplored beyond classification tasks. Approaches like the Forward-Forward algorithm rely on contrastive objectives, which fundamentally limit them to classification tasks and make adaptation to generation non-trivial. DiffusionBlocks is the first principled framework to extend this idea to modern generative architectures.
The diffusion insight that makes it work
The core idea is elegant. In standard training, what each layer does along the way is left unspecified -- layers collectively "somehow coordinate" to connect inputs to outputs. DiffusionBlocks instead gives each block an explicit role: move the representation a little closer to the target than the previous block did.
This matches the essence of diffusion models, which progressively denoise a signal toward a clean output. The theoretical grounding comes from a known connection: residual connections in Transformers correspond to discretized steps of an ordinary differential equation (ODE) -- the same class of equations underlying diffusion models. From there, the block-wise dynamics can be cast as the reverse (denoising) process of a diffusion model, giving each block a principled, independent training objective.

Once you see the network through this lens, each block handles one segment of the path from noise to target. No segment depends on another's output during training -- which is exactly what makes independent training possible.
Converting a network in three steps
The conversion from a standard Transformer to a DiffusionBlocks model requires just three modifications:
- Partition: Split the L layers into B blocks.
- Assign noise ranges: Each block is responsible for a range of "closeness to the target,