Skip to main content
Back

NVIDIA's Dory Reuses 20 Layers to Hit 96 Effective Layers at 4B

NVIDIA's new 4B model reuses a shared block of 20 layers up to four times, trading compute for accuracy without retraining.

  • NVIDIA released Dory1.0-4B, a looped LLM with 8 input, 20 recurrent, and 8 output layers.
  • Single checkpoint runs at 1, 2, or 4 loops to trade compute for accuracy, no retraining needed.
  • SWE-bench Verified climbs from 50.3 to 61.8 as loop count rises from 1 to 4.
  • Last-loop KV caching keeps memory footprint equal to a non-looped model regardless of loops.
  • Pre-trained on 6T tokens, mostly in NVIDIA's native 4-bit NVFP4 format, with 256K context.
  • Available under permissive OpenMDW-1.1 license, with vLLM support in review.

NVIDIA has released Dory1.0-4B, a 4-billion-parameter language model whose inference depth can change without retraining. Its shared 20-layer block can run one, two, or four times over each token, allowing the same weights to spend more computation on harder tasks.

Conventional transformers apply every layer once per token, fixing their depth when trained. Looped transformers reuse a group of layers, making effective depth a serving parameter. Dory therefore supports several compute profiles from one checkpoint, although additional loops increase arithmetic and latency per token.

Twenty Layers, Applied Again

Dory divides its network into three groups: an eight-layer Input block, a 20-layer Recurrent block, and an eight-layer Output block. The Input block’s representation is reinjected during every recurrent pass, and each pass uses the same 20 layers and parameters.

Recurrent passes Effective layer applications NVIDIA’s suggested use
1 36 Lower per-token compute
2 56 Default deployment
4 96 Agentic and coding workloads

Because the recurrent passes share weights, the parameter count remains 4 billion even when the model performs the equivalent of 96 layer applications.

Diagram showing Dory's input, recurrent, and output layer blocks
Dory reuses its 20-layer recurrent block between fixed input and output blocks.

The backbone uses normalized GPT, or nGPT, an architecture built around normalized parameter and activation representations. It interleaves sliding-window attention, which reads a local span efficiently, with full attention layers that can exchange information across the sequence. The model supports contexts up to 262,144 tokens, though prefill time and cache memory still grow with sequence length.

Keeping Recurrence Out of the Cache Bill

During autoregressive generation, an inference server stores attention keys and values for earlier tokens in a KV cache. A basic recurrent implementation keeps separate cached states for every pass because each pass produces different hidden representations, causing recurrent cache memory to grow with the loop count.