Skip to main content
Back

Zyphra unveils Zamba2-7B: Open-source LLM beats Mistral, Gemma

READ MORE

October 15, 2024 5 min read

Hey,

Welcome to today's edition of AlphaSignal, a newsletter for developers by developers.

We identify and summarize the top 1% news, papers, models, and repos in the AI industry. 

IN TODAY'S SIGNAL

Read time: 5 min 16 sec

🎖️ Top News


📌 Galileo

⚡️ Trending Signals


🛠️ Top of Github

  • mlc-llml: Universal LLM Deployment Engine with ML Compilation.

  • surya: OCR, layout analysis, reading order, table recognition in 90+ languages.

  • TEN-Agent: world’s first real-time multimodal agent integrated with the OpenAI Realtime API, RTC, and features weather checks, web search, vision, and RAG capabilities.


🧠 Lecture

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

Hybrid-SSM model

Zyphra Releases Zamba2-7B: A Hybrid SSM-Attention Model Outperforming Current 7B Leaders

⇧ 965 Likes

What's New

Zyphra has unveiled Zamba2-7B, a 7 billion parameter language model that combines Mamba2 blocks with shared attention layers. This open-source model surpasses other leading models in the ≤8B parameter range, including Mistral-7B, Gemma-7B, and Llama3-8B, in both quality and inference speed.

Key highlights

  • Outperforms Mistral-7B, Gemma-7B, and Llama3-8B in benchmarks

  • 25% faster time to first token, 20% higher tokens per second than Llama3-8B

  • Trained on 3 trillion tokens with 100B token "annealing" phase

  • Open-source under Apache 2.0 license

  • Instruct-tuned version available


Architecture 

  • The Zamba2-7B architecture features a backbone of Mamba2 layers interspersed with two shared attention layers in an ABAB pattern.

  • This design allocates more parameters to the Mamba2 backbone while preserving cross-sequence dependencies.

  • LoRA projectors applied to shared MLPs allow for depth-specific specialization.

Zamba2-7B Inference Performance

  • SOTA inference efficiency, including latency, throughput and memory usage.

  • Roughly 4 times the throughput of an equal-parameter transformer block.

  • One of the key reason for such performance is that Mamba blocks only have small hidden states to store and don't require a KV-cache. The model only stores KV states for shared attention block invocations, reducing memory usage.


Training Details

  • The model's training involved a 3 trillion token dataset, followed by a 100 billion token "annealing" phase using high-quality data. This approach contributes to its strong performance on a per-training-token basis and across standard benchmarks for the ≤8B parameter class.

  • The training process utilized 128 H100 GPUs over approximately 50 days, employing Zyphra's internal framework built on Megatron-LM. This demonstrates the feasibility of pushing the 7B scale frontier with moderate resources.


Access and availability
Developers can access Zamba2-7B through Huggingface and also from NVIDIA’s NIM site. The model is designed for on-device use and consumer GPUs, making it accessible for various applications.

Read more

Learn from Nvidia, HP, Databricks, and more

Nvidia, HP, Databricks, ServiceNow, and Twilio just announced the premier conference for GenAI:

  • Designing GenAI stacks for enterprise scale

  • AI governance, evaluation, and observability strategies

  • Production-ready GenAI app best practices

Developers can now access these key learnings to enhance their AI implementations and workflows.

Free registration

partner with us

TRENDING SIGNALS

Video Generation

Adobe introduces Firefly Video Model, enabling AI-generated video creation in Premiere Pro

⇧ 1,727 Likes

Multimodal Model

Rhymes AI releases Aria: open-source multimodal MoE model outperforming models 7x its size

⇧ 1,400 Likes

LLM

Researchers reveal LLMs' mathematical reasoning limitations, highlighting need for robust evaluation in production

⇧ 560 Likes

Perplexity

Perplexity introduces interactive chart feature in code interpreter, enhancing data visualization capabilities

⇧ 1,000 Likes

Google

Google introduces new Gemini API parameters, enhancing control and reproducibility

⇧ 1,200 Likes

TRENDING REPOS

LLM Deployment

mlc-llml

☆ 18,910 Stars

This repo helps you deploy LLMs across diverse hardware: GPUs (NVIDIA, AMD, Apple, Intel), CPUs, and web browsers. It uses ML compilation to optimize performance. You can run models like Llama 2 and Vicuna on your chosen device with OpenAI-compatible APIs.

OCR

surya

☆ 11,200 Stars

helps you do OCR, text detection, layout analysis, reading order detection, and table recognition for documents. It works in 90+ languages with high accuracy. You can process PDFs and images. It outperforms Tesseract in text detection (82.1% vs 63.1% precision) and matches Google Cloud Vision in OCR accuracy (97% similarity). The models use modified EfficientViT and Donut architectures, trained on diverse datasets. You can install it with pip and use it via CLI, Python API, or a Streamlit GUI.

Code Manager

TEN-Agent

☆ 711 Stars

helps you build high-performance multimodal AI applications with real-time interactions. You can integrate OpenAI's Realtime API, RTC, and custom extensions in C++, Go, or Python. It supports edge-cloud deployment, multi-language processing, and real-time agent state management. The drag-and-drop interface simplifies complex AI system creation, combining audio-visual tools, databases, and RAG.

LECTURE

AI Evolution

Andrej Karpathy on AI evolution: from self-driving cars to AGI

⇧ 5,600 Likes

Andrej Karpathy, former Tesla Autopilot leader and OpenAI founding member, shares insights on AI's evolution in a No Priors podcast interview. He discusses the shift from traditional programming to "Software 2.0", where neural networks and data define software behavior. 


He talks about on the importance of extremely smaller-sized distilled models where even 1Bn param model should be useful enough.


Karpathy explores vision-based approaches in self-driving cars, drawing parallels between autonomous vehicles and potential AGI development. He emphasizes data's crucial role in modern software and AI's impact on various sectors.


In this discussion, you'll learn about

  • Parallels between human cognition and AI models

  • Building high performance small models

  • Software 2.0 concept and its implications

  • Vision-based AI systems in autonomous driving

  • Comparison of Tesla and Waymo approaches

  • AGI development parallels with self-driving tech

  • AI's economic impact, particularly on transaction costs

  • Data's role as the new 'code' in AI-driven software

  • AI's potential in reshaping education and productivity

Watch it

Thank You.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Advertise