Skip to main content
Back

DeepSeek Drops V4-Pro-Base Open Weights Behind Its 1.6T Flagship Model

DeepSeek drops the raw base weights for its 1.6T-parameter V4-Pro, unlocking fine-tuning on the strongest open-source foundation model built to date

Model, Full Access
  • Base weights released: DeepSeek drops the raw pre-trained checkpoint for DeepSeek-V4-Pro-Base, a 1.6T-parameter MoE model, enabling fine-tuning and alignment research.
  • Strongest open foundation model: V4-Pro-Base outperforms V3.2-Base and V4-Flash-Base across knowledge, reasoning, coding, and long-context benchmarks.
  • Radical attention efficiency: New CSA+HCA hybrid attention cuts KV cache to 10% and inference FLOPs to 27% of V3.2 at 1M-token context.
  • Trained on Huawei silicon: V4 is the first frontier model confirmed to run on Huawei Ascend 950PR chips, bypassing NVIDIA entirely for training.
  • Muon optimizer at scale: DeepSeek replaces AdamW with the Muon optimizer across 33 trillion training tokens -- the largest production deployment of Muon to date.
  • MIT license, massive hardware bar: Weights are free and commercially usable, but the 1.6TB model file requires serious cluster infrastructure to self-host.

DeepSeek-V4-Pro-Base just landed on Hugging Face, releasing the unaligned pre-trained weights behind DeepSeek's flagship V4-Pro model. This is the raw foundation before any instruction tuning or reinforcement learning from human feedback has been applied -- the kind of checkpoint researchers and fine-tuning teams actually want. It joins the already-released instruct variant and completes the full V4-Pro model family.

The model behind the weights

DeepSeek-V4-Pro is a Mixture-of-Experts (MoE) language model with 1.6T total parameters and 49B activated per token, supporting a context length of one million tokens. MoE means the model has a large pool of specialized sub-networks ("experts"), but only a small fraction of them activate for any given token -- so you get the knowledge capacity of a 1.6T model at roughly the compute cost of a 49B dense one.

The V4 series is DeepSeek's first two-tier lineup and introduces a new architecture -- the first from the lab since V3. DeepSeek-V4-Pro-Base demonstrates a decisive leap in capability, establishing near-universal dominance over both DeepSeek-V3.2-Base and DeepSeek-V4-Flash-Base. With improvements across almost all categories, it reaches new performance highs among DeepSeek base models on the most demanding benchmarks. On knowledge-intensive evaluations, it delivers dramatic gains, while also substantially advancing long-context understanding. On most reasoning and code benchmarks, DeepSeek-V4-Pro-Base also exceeds both previous models -- confirming it as the strongest foundation model in the DeepSeek series.

Three architectural bets that paid off

V4's architecture is a meaningful departure from V3. Three innovations do the heavy lifting:

  1. Hybrid Attention (CSA + HCA). V4 introduces a two-layer hybrid: Compressed Sparse Attention (CSA) handles long-range token selection via learned sparsity patterns, while Heavily Compressed Attention (HCA) applies extreme rank reduction to the local-window cache. The combined effect is that V4-Pro running a 1M-token prompt consumes 27% of the FLOPs and 10% of the KV cache memory required by V3.2 at the same context length. That is not a marginal improvement -- it is the difference between 1M context being a demo and being a production feature.
  2. Manifold-Constrained Hyper-Connections (mHC). mHC replaces conventional residual connections -- the plumbing that lets gradients flow cleanly through deep networks. At trillion-parameter scale these can become unstable. DeepSeek's report describes mHC as a reinforcement of residual pathways that keeps signal propagation stable without giving up expressivity. It is the kind of change that does not move benchmarks dramatically on its own, but makes the rest of the training run possible.