Skip to main content
Back

New technique allows open-source LLM to beat top closed-source models

READ MORE

September 6, 2024 6 min read

Hey,

Welcome to today's edition of AlphaSignal, a newsletter for developers by developers.

We identify and summarize the top 1% news, papers, models, and repos in the AI industry. 

IN TODAY'S SIGNAL

Read time: 6 min 38 sec

🎖️ Top News

  • Matt Shumer releases Reflection 70B, an open-source model thats better than Llama 3 and GPT4-o, 405B coming next week.


📌 Ray Summit

  • Take your AI/ML skills to the next level at Ray Summit and join AI experts in San Francisco on September 30, 2024.

⚡️ Trending Signals


📌 Growth |School


🧠 Top Papers

  • OLMoE: Open-source MoE LLM achieving SOTA performance with efficient parameter usage.

  • In Defense of RAG in the Era of Long-Context Language Models.

  • FLUX that Plays Music: Scalable text-to-music generation using rectified flow Transformers.


🧠 Deep Dive

  • Understanding how the new vLLM, the most popular LLM inferencing library is improving throughput by 2.7x and reducing latency by 5x.

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

LLM

Reflection 70B, new open-source LLM surpasses Claude 3.5 Sonnet and GPT-4o on key benchmarks

⇧ 8,348 Likes

What's New

Matt Shumer releases a new LLM, Reflection 70B, the world’s top open-source model. It is trained using Reflection-Tuning, a technique to help LLMs to fix their own mistakes. 405B version coming next week.


Performance and Benchmarks

  • Tops MMLU, MATH, IFEval, and GSM8K benchmarks

  • 89.9% MMLU, 79.7% MATH, 90.1% IFEval

  • Exceeds GPT-4o across all tested metrics

  • Achieves over 99% accuracy on GSM8K

  • Significantly outperforms Llama 3.1 405B

Reflection-Tuning Process

  • The technique separates planning into a distinct step, enhancing chain-of-thought reasoning and producing concise outputs. 

  • This approach addresses hallucination tendencies in current LLMs by implementing a self-correction mechanism before committing to an answer.

  • Here's how it works

    1. During inference, the model generates reasoning within <thinking> tags.

    2. If it detects a mistake, it uses <reflection> tags to correct itself before moving forward.

    3. Only after this self-correction does the model provide a final answer, enclosed in <output> tags.

Development and Availability

  • Created using Glaive AI's data, Reflection 70B was developed over several weeks. 

  • The 70B model weights are accessible on Hugging Face. 

  • A 405B version, expected to further outperform current leading models, is scheduled for release next week.

Technical Considerations

  • The model has undergone decontamination against multiple benchmarks using LMSys' LLM Decontaminator. Early evaluations show strong performance on tasks like LaTeX error correction and basic counting. 

  • However, the reflection process increases token usage, potentially impacting response speed. 

  • A comprehensive report detailing the process and findings is forthcoming. 

  • The team plans to continue refining the technique, with hints at additional improvements in future iterations.

  • No, success on an 8B scale yet.

Read more

Take Your AI/ML Skills to the Next Level at Ray Summit

Get an in-person, intensive day of training at Ray Summit. Enhance your AI toolkit and get everything you need to handle complex AI projects. 


Join AI experts and thought leaders in San Francisco on September 30, 2024. 


Tickets will sell out, so grab yours today. AlphaSignal readers get 15% off.

Attend the summit

partner with us

TRENDING SIGNALS

AI Agent

Replit launches AI agent capability that codes and deploys full apps from prompts

⇧ 10,766 Likes

Multi Modal AI

GroqCloud started supporting multi-modality with image, audio & text

⇧ 533 Likes

LLM

Anthropic introduces Claude for Enterprises with  500K context window, native GitHub integration and higher security

⇧ 1,700 Likes

AI for Coding

New research finds the real impacts of AI on programmer productivity, a 26.08% increase in completed tasks

⇧ 2,026 Likes

LLM

DeepSeek releases DeepSeek-V2.5 with enhanced writing, instruction-following, and human preference alignment, available on Web and API

⇧ 389 Likes

Use AI to 10x your productivity and efficiency at work

Struggling to balance work and time?  This 3-hour AI & ChatGPT workshop (usually $399) is FREE for the first 100 AlphaSignal readers.


Discover AI strategies to boost productivity, automate tasks, and grow your business 10x faster.

👉 Save your spot now (seats are filling fast)

TOP PAPERS

LLM

OLMoE: Open Mixture-of-Experts Language Models

⇧ 1,590 Likes

Problem
LLMs face trade-offs between performance and cost. Mixture-of-Experts (MoE) models offer a solution, but lack open-source options. OLMOE-1B-7B addresses this gap with 6.9B total parameters, activating only 1.3B per token.


Solution
OLMOE-1B-7B, a fully open-source MoE LLM, was trained on 5.1 trillion tokens. It uses 64 small experts per layer, with 8 activated. The model employs dropless token-choice routing and auxiliary losses for stability.


Results
OLMOE-1B-7B outperforms models with similar active parameters, matching Llama2-13B's MMLU score (54.1%). After instruction tuning, it surpasses Llama2-13B-Chat and DeepSeekMoE-16B on benchmarks like MMLU, GSM8k, and HumanEval.

RAG

In Defense of RAG in the Era of Long-Context Language Models

⇧ 1,180 Likes

Problem
Recent studies favor long-context LLMs over Retrieval-Augmented Generation (RAG) for question-answering tasks. However, this paper argues that extremely long contexts can diminish focus on relevant information, potentially degrading answer quality.


Solution
The authors propose Order-Preserve RAG (OP-RAG), which maintains the original order of retrieved chunks in the context, unlike traditional RAG that orders by relevance.


Results
OP-RAG outperforms long-context LLMs and vanilla RAG. Using Llama3.1-70B, OP-RAG achieves 47.25 F1 score on EN.QA with 48K tokens, compared to 34.26 F1 score with 117K tokens for long-context LLM without RAG. On EN.MC, OP-RAG reaches 88.65% accuracy with 24K tokens, surpassing Gemini-1.5-Pro's 85.57% with 188K tokens.

Transformers

FLUX that Plays Music

⇧ 1,500 Likes

Problem

Existing text-to-music generation models struggle with efficiency and quality. Current approaches use language models or diffusion models to represent quantized waveforms or spectral features, but face computational challenges and limited scalability.


Solution

FluxMusic introduces a rectified flow Transformer architecture for text-to-music generation in latent VAE space. It uses double and single stream attention blocks, multiple pre-trained text encoders, and a modulation mechanism for coarse textual information.


Results

FluxMusic outperforms existing models on automatic metrics and human evaluations. The giant version (2.1B parameters) achieves state-of-the-art results: FAD 1.43, KL 1.25, IS 2.98, and CLAP 0.36 on MusicCaps dataset. Human evaluations show improved overall quality and text relevance compared to AudioLDM 2 and MusicGen.

TOP LECTURE

LLM Inferencing

How the new vLLM achieves 2.7x higher throughput and 5x faster time per output token

⇧ 349 Likes

vLLM is the one of the most popular for LLM inferencing library. It is known for being high-throughput and memory-efficient inference and serving engine for LLMs. Today they just released their next version v0.6.0.

vLLM v0.6.0 boosts LLM inference performance significantly

  •  2.7x higher throughput for Llama 8B

  •  5x faster time per output token for Llama 8B

  •  1.8x higher throughput for Llama 70B

  •  2x less time per output token for Llama 70B

Key enhancements

  • Separated API server and inference engine. By eliminating GIL (Global Interpreter Lock) constraints, both components can operate more efficiently without CPU contention, leading to improved performance.

  • Multi-step batch scheduling

  • Asynchronous output processing

  • Object caching and non-blocking data transfers (24% throughput improvement with this)

Results

  • Highest throughput on H100 GPUs for certain workloads

  • Comparable latency to TensorRT-LLM and SGLang

Overall, now you get faster, more efficient LLM inference without sacrificing features or usability.

Read more

Thank You.

Want to promote your company, product, service, or event to 190,000+ AI researchers and engineers?

Sponsor