Skip to main content
Back

New VPTQ technique enables 70B LLM on single GPU

READ MORE

October 8, 2024 4 min read

Hey,

Welcome to today's edition of AlphaSignal, a newsletter for developers by developers.

We identify and summarize the top 1% news, papers, models, and repos in the AI industry. 

IN TODAY'S SIGNAL

Read time: 4 min 39 sec

🎖️ Top News


📌 Oxylabs

⚡️ Trending Signals


🛠️ Trending Repos

  • manim: Python framework for creating mathematical animations.

  • agents: Build real-time multimodal AI applications.

  • Python: All Algorithms implemented in Python


🧠 Recommended Lecture

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

LLM Research

New Research Vector Post-Training Quantization Achieves 2-Bit Compression for LLMs

⇧ 986 Likes

What's New

Vector Post-Training Quantization (VPTQ) is a new technique that compresses large language models to extremely low bit-widths without retraining.


It addresses the challenge of deploying massive models on resource-constrained devices by using vector quantization instead of traditional scalar-based weight quantization.


Key highlights

  • Compresses 405B parameter models to 1-2 bits in ~17 hours

  • Enables 70B parameter model inference on a single RTX 4090 GPU

  • Outperforms GPTQ, QuIP#, and AQLM in perplexity and accuracy

  • Maintains high inference throughput with low quantization overhead

  • Supports PyTorch, TensorFlow, and Hugging Face Transformers

  • Experiments demonstrate reduced quantization perplexity and increased accuracy and inference throughput across benchmarks like LLaMA-2, Mistral-7B, and LLaMA-3 compared to state-of-the-art methods.


Core innovation

  • VPTQ's core innovation lies in its use of Second-Order Optimization and Channel-Independent Second-Order Optimization.

  • This approach minimizes quantization errors and allows for more efficient compression.

  • The method compresses weight vectors into indices using lookup tables, achieving better accuracy at 1-2 bits compared to other methods.


Performance metrics

  • Performance metrics demonstrate VPTQ's effectiveness. For the LLaMA-2 70B model quantized to 2.07 bits, VPTQ achieves a perplexity of 3.93 on WikiText-2, an inference speed of 9.7 tokens/second, and memory usage of 19.54 GB.

  • These results indicate minimal loss in model quality while significantly reducing computational requirements.


The scalability of VPTQ is evident in its ability to handle models up to 405B parameters. This suggests potential applicability to even larger models, though specific performance on architectures beyond the LLaMA family remains to be explored. The method's impact on downstream task performance compared to full-precision models is an area for further investigation.

Read more

First AI copilot for building web scrapers

OxyCopilot, a new AI-powered tool from Oxylabs, helps developers build scalable web scrapers and parsers. It identifies complex website patterns, including nested and listed data, enabling data extraction without coding.


This free tool streamlines the scraping process, allowing developers to focus on results.


With the help of OxyCopilot, you can: 

  • Parse raw HTML into structured data 

  • Adapt to web layout changes in seconds

  • Free up your time and focus on results

Test for free

partner with us

TRENDING SIGNALS

Andrej Karpathy

Karpathy reveals unexpected convergence in LLM outputs, sparking debate on fine-tuning diversity

⇧ 6,200 Likes

Diffusion Model

Researchers unveil LOTUS: open-source diffusion-based visual foundation model for high-quality dense prediction

⇧ 1,025 Likes

Computer Vision

New study introduces Posterior-Mean Rectified Flow, enhancing photo-realistic face restoration with optimal MSE

⇧ 1,400 Likes

Prompt Engineering

An Open-source Github repository releases 1,405 jailbreak prompts.

⇧ 1,401 Likes

Computer Vision

New open-source python package, supervision allows developers to implement line crossing detection in 30 minutes

⇧ 1,000 Likes

TRENDING REPOS

Math Animation

manim

☆ 21,600 Stars

Manim helps you create programmatic math animations. It's a Python library for generating explanatory videos, like those on 3Blue1Brown. You can render complex mathematical concepts visually. It supports both 2D and 3D animations. The community version offers frequent updates and extensive documentation.

Agents

agents

☆ 2,600 Stars

Helps you build AI agents that process and generate real-time text, audio, video, and images via LiveKit sessions. It integrates popular LLMs, STT, TTS services, and RAG databases. You can create voice assistants with turn detection, interruption handling, and function calling. It supports local, self-hosted, and cloud deployments with load balancing.

Algorithm

Python

☆ 190,000 Stars

Helps you learn and implement algorithms in Python. You'll find code for various domains like computer vision, data structures, and machine learning. It's community-driven, with 100+ contributors. The codebase follows black formatting and uses pre-commit hooks for quality control.

LECTURE

Prompt Caching

Anthropic prompt caching: optimize API calls for faster LLM interactions.

⇧ 5100 Likes

Prompt Caching is a new API feature that optimizes large language model interactions. It caches and reuses consistent parts of prompts, reducing processing time and costs for repetitive tasks.


This technique is particularly useful for scenarios involving large contexts, multiple examples, or long conversations. Prompt Caching works by checking if a prompt prefix is already cached from a recent query and using it if found, otherwise processing and caching the full prompt.


You'll learn about:

  • Implementing Prompt Caching with the Messages API

  • Structuring prompts for effective caching

  • Cache limitations and supported models

  • Tracking cache performance

  • Pricing structure for cached content

  • Best practices for optimizing cache usage

  • Troubleshooting common issues

  • Privacy and data separation in caching

Read more

Thank You.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Advertise