DeepSeek-VL Matches GPT-4V Reading Charts and PDFs at 7B Parameters
DeepSeek-VL-7B-Chat is a free, open-source vision-language model that processes 1024×1024 images with near-GPT-4V accuracy on real-world tasks
Takeaways
- Open-source VLM at 7B parameters that handles 1024×1024 images commercially for free under the DeepSeek license.
- Hybrid vision encoder combines SigLIP-L (semantics) and SAM-B (fine-grained detail) to avoid CLIP's blind spots on dense text and small objects.
- Near-GPT-4V performance on SEEDbench (70.4 vs 71.6) and beats most open-source 7B/13B models on OCR, image comprehension, and hallucination benchmarks.
- Preserves language ability through a 70/30 text-to-image training ratio and a modality warm-up strategy, scoring 68.4 on HellaSwag vs. 68.5 for the text-only base model.
- Weak on math reasoning: GSM8K drops from 55.0 (text-only) to 18.0 after multimodal training, a known trade-off at this scale.
- Available now on Hugging Face with a GitHub repo and full inference code.
DeepSeek-VL-7B-Chat is an open-source vision-language model (VLM) built to handle the messy, high-resolution inputs that show up in real-world applications: scanned PDFs, dense charts, web screenshots, handwritten formulas, and natural photos. It sits at 7 billion parameters, supports 1024×1024 image resolution, and is free to use commercially under the DeepSeek model license.
The problem it was built to solve
Most open-source VLMs at the time of this model's release were trained primarily on academic benchmark datasets and operated at low resolutions, typically 336×336 or 448×448 pixels. That works fine for simple image captioning, but falls apart when you need to read small text in a document, parse a financial chart, or understand a UI screenshot. The DeepSeek team structured their approach around data that covers real-world scenarios including web screenshots, PDFs, OCR, charts, and knowledge-based content.
There was also a subtler problem: multimodal training tends to destroy language ability. If you train a language model on too many image-text pairs, it starts forgetting how to do pure text tasks. DeepSeek-VL directly addresses this with a novel training strategy.
A hybrid vision encoder that punches above its resolution
The key architectural innovation is a hybrid vision encoder that combines two separate models to handle images at different levels of detail:
- SigLIP-L at 384×384: a text-aligned encoder that extracts high-level semantic meaning from a lower-resolution view of the image.
- SAM-B (Segment Anything Model) at 1024×1024: a vision-only encoder that captures fine-grained spatial detail, pixel-level texture, and dense text.
This hybrid encoder efficiently processes high-resolution images (1024×1024) while maintaining a relatively low computational overhead, ensuring the model's ability to capture critical semantic and detailed information across various visual tasks. The two encoders are fused through a two-layer MLP adaptor, and their combined output is compressed into just 576 visual tokens before being passed to the language model. That token budget keeps inference costs manageable even for multi-turn conversations.
Why does this matter? Standard CLIP-family encoders like SigLIP alone suffer from what the paper calls "CLIP-blind pairs": visually distinct images that get encoded as nearly identical because the encoder is optimized for semantic similarity, not pixel-level discrimination. Adding SAM fixes that blind spot.