Skip to main content
Back

DeepSeek Unveils New Open-Source GPT-4o Rival

TRY NOW

December 27, 2024 4 min read

Hey,

Welcome to AlphaSignal – the most read newsletter by AI developers. 


We bring you the top 1% of news, papers, models, and repos, all summarized to keep you updated on the latest in AI.

IN TODAY'S SIGNAL

Read time: 4 min 33 sec

🎖️ Top News

⚡️ Trending Signals

📝 Top Papers

  • CosyVoice 2 reduces streaming delays and improves TTS accuracy, enhancing user experience.

  • Byte Latent Transformer scales efficiently with byte-level encoding, outperforming tokenized models.

  • Vision Transformers use register tokens to eliminate artifacts and boost visual task performance.

🧠 Python Tip

  • Gitingest turns GitHub repos into LLM-ready text for faster code analysis.

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

LLM

DeepSeek announces DeepSeek-V3, an open-source model outperforming GPT-4o and Claude Sonnet 3.5 in coding, math

⇧ 20,539 Likes

What's New

DeepSeek-V3 is a fully open-source 671 billion parameter Mixture-of-Experts (MoE) language model that activates only 37 billion parameters per token. This design reduces computational requirements while delivering strong task performance. The model uses 14.8 trillion tokens for training, making it one of the most efficient large-scale models available.

Key Technical Features

  • DeepSeek-V3 is trained in just 2.8 million GPU hours which is significantly less than comparable models.

  • Auxiliary-loss-free load balancing: This technique optimizes computation by distributing the workload evenly across MoE layers without introducing additional complexity.

  • MoE architecture: DeepSeek-V3 uses a MoE architecture, where only a subset of parameters is activated per token. This improves computational efficiency by using fewer active parameters while maintaining model accuracy. The architecture dynamically activates different experts based on the input, optimizing resource usage.

  • FP8 mixed precision: The model trains with FP8 mixed precision, which reduces memory consumption and power usage by storing weights and activations in lower precision. This allows faster training with minimal impact on performance, making the model more energy-efficient.

  • Multi-Token Prediction (MTP): It processes multiple tokens simultaneously. This boosts throughput and enables faster processing without adding latency, allowing the model to handle larger batches and increase efficiency during inference.

  • 14.8 trillion tokens: This helps in multilingual and domain-specific learning.

Performance Benchmarks

In the HumanEval Pass@1 benchmark, DeepSeek-V3 scores 65.2%, outperforming Claude Sonnet 3.5. It also excels in multilingual benchmarks like XSum and TyDi QA, delivering competitive results compared to GPT-4o and LLaMA 3. For coding, math, and reasoning tasks, it demonstrates superior efficiency and precision.

Access and Availability

You can access DeepSeek-V3 on Hugging Face. Test it via API or a free chat interface. The API pricing stays fixed and minimal until February 8, 2024, giving you time to integrate it into your workflows.

Try now

TRENDING SIGNALS

Reasoning Model

Qwen unveils QvQ, an open-weight visual reasoning model, outperforms GPT4o, Claude Sonnet 3.5 in MMMU and MathVista 

⇧ 6,582 Likes

LLM

Meta introduces Large Concept Models (LCM), a model that processes concepts instead of tokens, trained on 7.7T tokens

⇧ 3,524 Likes

Coding Assistant

Cognition updates Devin, its AI software engineer, with full API support, 10% faster performance and custom workflows

⇧ 835 Likes

Model Deployment

Hugging Face enables direct use of private GGUFs in Ollama, simplifying custom models, fine-tunes, and quants

⇧ 629 Likes

Cloud AI Tool

Microsoft Research releases AIOpsLab, an open-source framework for testing and improving AI agents in cloud operations

⇧ 914 Likes

TOP PAPERS

Text-to-Speech

CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Model

⇧ 692 Likes

Problem

CosyVoice was introduced by Alibaba to create high-quality speech synthesis, but it struggled with response delays and real-time processing, especially for streaming speech. These issues made it difficult to provide smooth, interactive experiences.

Solution
CosyVoice 2 improves speed and accuracy by optimizing how speech tokens are used, simplifying its model, and adding a new method to handle both streaming and non-streaming synthesis.

Results
CosyVoice 2 reduces delays, lowers pronunciation errors, and improves stability, resulting in better overall speech quality and more control over emotional tone and accents.

Model Architecture

Byte Latent Transformer: Patches Scale Better Than Token

⇧ 3,525 Likes

Problem

Tokenization-based LLMs face challenges in scaling efficiently while maintaining performance. Traditional models rely on fixed vocabularies and tokens, limiting their inference efficiency and robustness.

Solution
The Byte Latent Transformer (BLT) introduces byte-level encoding with dynamically sized patches, improving efficiency. Patches are allocated based on byte entropy, using more compute where data complexity is higher. The model scales up to 8B parameters and 4T training bytes.

Results
BLT outperforms tokenization-based models in scaling efficiency, offering better performance with lower inference costs. It also improves reasoning and long-tail generalization, showing that byte-level models can scale effectively without fixed vocabularies.

Transformers

Vision Transformers Need Registers

⇧ 4,793 Likes

Problem

Vision Transformers (ViTs) exhibit artifacts in feature maps, especially in low-informative background areas of images. These artifacts arise from high-norm tokens repurposed for internal computations, leading to issues in downstream visual processing and attention maps.

Solution
The paper proposes adding additional "register tokens" to the input sequence of ViTs to fill the role of storing, processing, and retrieving global information. This approach resolves the artifact problem in both supervised and self-supervised models, improving feature map smoothness and attention map clarity.

Results
This solution eliminates the artifacts, sets a new state of the art for self-supervised visual models, and improves performance on dense visual prediction tasks. The introduction of register tokens enables more interpretable attention maps, leading to better overall model performance.

PYTHON TIP

Turn GitHub Repositories into LLM-Ready Text with Gitingest

When working with code from GitHub repositories, it can be time-consuming to manually parse through files for relevant information. Gitingest simplifies this by turning any GitHub repository into a text digest optimized for LLMs. This provides an easy way to analyze codebases without having to clone or download files.

Applications

Use Gitingest for quick code context analysis, summarizing large repositories, or analyzing code structure before diving into implementation details.

You can use it in the CLI or import the Python package into your code by running the following code:


from gitingest import ingest

summary, t, content = ingest("path/to/directory")

#or from URL
summary, t, content = ingest("https://github.com/cyclotruc/gitingest")

At Alpha Signal, our mission is to build a sharp, engaged community focused on AI, machine learning, and cutting-edge language models, helping over 200,000 developers stay informed and ahead. We’re passionate about curating the best in AI, from top research and trending technical blogs to expert insights and tailored job opportunities. We keep you connected to the breakthroughs and discussions that matter, so you can stay in the loop without endless searching. We also work closely with partners who value the future of AI, including employers and advertisers who want to reach an audience as passionate about AI as we are.


Our partnerships are based on shared values of ethics, responsibility, and a commitment to building a better world through technology.Privacy is a priority at Alpha Signal. Our Privacy Policy clearly explains how we collect, store, and use your personal and non-personal information. By using our website, you accept these terms, which you can review on our website. This policy applies across all Alpha Signal pages, outlining your rights and how to contact us if you want to adjust the use of your information. We’re based in the United States. By using our site, you agree to be governed by U.S. laws.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Work with us