Soup Trains Llama-3.1-8B on a 4 GB Laptop GPU for Free
Layer streaming plus 4-bit quantization pins an 8B model into 3.32 GB of VRAM, turning laptop GPUs into real fine-tuning boxes.
Takeaways
- Soup fine-tunes an 8B LLM on a 4 GB laptop GPU at 119.6 tok/s, 3.32 GB peak.
- Layer streaming keeps the frozen base in RAM, sends one decoder layer at a time to VRAM.
- Bit-exact against resident training, replicated on H100 at 113 tok/s same 3.32 GB.
- DPO, ORPO, SimPO, KTO stream too, with the reference model costing no extra weights.
soup shipnow includes a noise-floor and benign-prompt detector to catch gate gaming.- Install:
pip install "soup-cli[train]", Apache-2.0, Python 3.10 to 3.12, Zenodo paper.
Fine-tuning a text model that would not otherwise fit into your GPU used to mean paying for a rented H100 or wrestling with sharding across multiple cards. A new open-source CLI called Soup collapses that workflow into a single YAML file, and its headline trick is a technique the author calls layer streaming, which trains an 8B model on a 4 GB laptop GPU.
The pitch is unusually concrete: Llama-3.1-8B-Instruct plus NF4 quantization runs at 119.6 tokens per second with a 3.32 GB peak on an RTX 3050 Laptop 4 GB, bit-exact against a normal resident run, and reproduced independently on an H100 at 113.00 tok/s in the same 3.32 GB. Everything is Apache-2.0, offline, and driven from one command.
Streaming layers instead of loading them
The idea is simple to state. The frozen base model stays in host RAM, and a dedicated CUDA stream feeds it to the GPU one decoder layer at a time, so peak VRAM is bounded by a single layer rather than the whole model. Only the LoRA adapter, the current layer, and a small double-buffer live on the card.
Combine that with 4-bit NF4 quantization of the streamed base, which shrinks the store roughly fourfold, and an 8B model fits inside 3.32 GB with room to spare. In the config, it is a single opt-in key:
training:
stream_layers: true # base streams out of VRAM; only the adapter trains
quantization: 4bit # NF4 store, ~4x smaller
batch_size: 4 # bigger batches amortise the weight read
stream_source: auto # RAM when it fits, NVMe when it does not