Skip to main content
Back

Meta Launches Mobile Llama 3.2: 4x Faster, 56% Smaller

READ MORE

October 25, 2024 6 min read

Hey,

Welcome to today's edition of AlphaSignal, a newsletter for developers by developers.

We identify and summarize the top 1% news, papers, models, and repos in the AI industry. 

IN TODAY'S SIGNAL

Read time: 6 min 58 sec

🎖️ Top News


📌 Assembly AI 

⚡️ Trending Signals


📌 Deep Atlas


🧠 Top Papers

🧠 Python Tip

  • Memory-efficient caching for deep learning using python's weakref module.

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

Multimodal Embedding Model

Meta Releases Mobile-Optimized Quantized Versions of Llama 3.2 Models for On-Device AI

⇧ 4,377 Likes

What's New

Meta has launched quantized versions of Llama 3.2 1B and 3B models designed to run directly on mobile devices, marking a shift from data center-dependent AI to on-device processing. These models enable direct deployment on phones and tablets while maintaining comparable performance to their larger counterparts.


Key Highlights

  • First-ever mobile-optimized quantized Llama models with 2-4x speedup.

  • Dual quantization methods: QLoRA for accuracy, SpinQuant for portability.

  • 56% smaller model size, 41% less memory usage vs BF16.

  • 8K token context support optimized for mobile constraints.

  • Direct deployment on Qualcomm/MediaTek SoCs with Arm CPUs.

  • ExecuTorch framework compatibility with full Hugging Face integration

Key Performance Metrics

  • 2-4x faster inference speed vs original BF16 format.

  • 56% reduction in model size.

  • 41% decrease in memory usage.

  • 2.5x improvement in decode latency.

  • 4.2x improvement in prefill latency


Technical Architecture

  • The quantization implementation combines two distinct approaches.

  • The primary method, Quantization-Aware Training with LoRA adapters (QLoRA), simulates quantization effects during training while using low-rank adaptation.

  • The secondary approach, SpinQuant, handles post-training quantization using rotation matrices learned from WikiText calibration data, offering a solution for scenarios without access to training datasets.

  • 8K token context length limitation.

Practical Implementations

  • Testing on Android devices (OnePlus 12, Samsung S24+, Samsung S22) demonstrates consistent performance across different hardware configurations.

  • iOS compatibility is confirmed for accuracy but lacks detailed performance benchmarks.

  • For developers, the implementation process integrates with existing tools through ExecuTorch and Llama Stack. The models support both direct deployment and further fine-tuning through LoRA adapters. 

Read more

Build the next big thing with conversational data

Speech AI now drives product strategy. The latest models deliver higher accuracy and faster performance to maintain competitive advantage.


AssemblyAI's Speech-to-Text API delivers

  • 95% accuracy rate

  • 30% reduction in hallucinations vs competitors

  • 35-second processing time for 63-minute audio files

  • Native language detection and speaker separation

Start building robust voice applications with AssemblyAI's API now.

Start building today

partner with us

TRENDING SIGNALS

Anthropic

Anthropic launches code execution in Claude, enabling direct data analysis and reproducible outputs

⇧ 6,466 Likes

Perplexity

Perplexity launches native macOS app, bringing AI search capabilities offline for developers

⇧ 1,800 Likes

LLM

Cohere For AI releases Aya Expanse: open-weights multilingual models outperforming existing language-specific LLMs

⇧ 390 Likes

Google

Google DeepMind releases MusicFX DJ, integrating real-time AI music generation into Labs platform

⇧696 Likes

AI Computers Interface

Open-Interpreter integrates Claude's computer control capabilities, enabling direct system interactions across multiple LLMs

⇧ 6,400 Likes

Learn AI/ML from first principles

Get hands-on machine learning experience with senior software engineers. Learn to ship ML-powered features in just weeks.


Now accepting applications from experienced developers for this technical deep-dive into practical ML implementation.

Apply Now

TOP PAPERS

LLM

Looking Inward: Language Models Can Learn About Themselves by Introspection

⇧ 1,140 Likes

Problem
LLMs have been assumed to only use training data for knowledge acquisition. This paper investigates whether LLMs can gain knowledge through introspection - accessing information about themselves not contained in or derivable from training data. They test if models can predict their own behavior better than other models trained on the same behavior data.

Results
After finetuning GPT-4, GPT-4o and Llama-3-70B on self-prediction tasks, models consistently outperform cross-prediction by other models (17% higher accuracy for Llama-70B vs GPT-4o). Models show calibrated predictions of their response distributions and can adapt predictions when their behavior changes through additional finetuning. However, introspection fails on complex tasks requiring longer outputs.


Solution
The researchers developed a self-prediction training framework where models predict properties of their hypothetical responses (e.g. "Would your output be even or odd?"). They compare self-prediction accuracy to cross-prediction by other models trained on the same behavior data. The models are tested on held-out tasks after training.

Computer Vision

You need between 200-600 images to train AI for copying an artist's style (e.g. Van Gogh's artistic style ).

⇧ 514 Likes

Problem
Text-to-image models trained on internet-scraped datasets can imitate copyrighted/private content, raising legal concerns. The paper studies how many training images of a concept are needed for a model to imitate it - the "imitation threshold".


Solution
The authors propose MIMETIC2, a method to estimate imitation thresholds without expensive model retraining. They analyze SD1.1, SD1.5 and SD2.1 models across human faces and art styles domains by comparing generated vs training images using domain-specific similarity metrics.


Results
The imitation threshold ranges between 200-600 images depending on model and domain - 364 images for celebrity faces (SD1.1/1.5), 527 for SD2.1. Art style thresholds are lower: 112-198 images (SD1.1/1.5), 185-241 (SD2.1). Results validate against human evaluation with 82.5-95% agreement.

Chain-of-Thought

A Theoretical Understanding of Chain-of-Thought: Coherent Reasoning and Error-Aware Demonstration

⇧ 487 Likes

Problem
LLMs demonstrate strong reasoning with Chain-of-Thought (CoT) prompting, but theoretical understanding remains limited. Previous analyses treat reasoning steps in isolation (Stepwise ICL) rather than as an integrated process. This fails to capture how real LLMs consider the full context when predicting next tokens.


Solution
The authors develop theoretical framework comparing Stepwise ICL to "Coherent CoT" where later steps integrate earlier reasoning. They analyze model sensitivity to errors in different reasoning steps and propose incorporating both correct and incorrect reasoning paths in prompts.


Results
Coherent CoT achieves better error correction and prediction accuracy than Stepwise ICL. Models are more sensitive to errors in intermediate reasoning steps vs final outcomes. Experiments across multiple datasets (BBH, GSM8K) and models (GPT-3.5/4, Gemini Pro, DeepSeek) show incorporating incorrect reasoning examples with explanations improves performance by up to 6.6%.

PYTHON TIP

Memory-efficient caching for deep learning using python's weakref module

You can store large tensors and intermediate results without worrying about memory leaks. The cache automatically cleans up when Python's garbage collector runs or when memory pressure increases.


Core Benefit


weakref lets you cache model outputs without holding strong references. When your original tensors get deleted, the cache entries disappear automatically.


Quick Implementation



import weakref
from typing import Dict, Any
import torch

class SmartCache:
    def __init__(self):
      self.cache: Dict[
          str,
          Any
      ] = weakref.WeakValueDictionary()
    
    def store(self, key: str, tensor: torch.Tensor) -> None:
        self.cache[key] = tensor
    
    def get(self, key: str) -> torch.Tensor:
        return self.cache.get(key)

# Usage Example
model_cache = SmartCache()

# Create a large tensor
big_tensor = torch.randn(1000, 1000)

# Cache it
model_cache.store('layer1_output', big_tensor)

# Use it later
cached_tensor = model_cache.get('layer1_output')

# When big_tensor goes out of scope, 
#cache entry auto-deletes
del big_tensor


Why This Helps
The WeakValueDictionary holds weak references to your tensors. When the original tensor's reference count hits zero, Python automatically removes it from the cache, freeing memory.


WeakValueDictionary prevents memory leaks by auto-removing cached tensors when they're no longer used, saving RAM during large model inference without manual cleanup.


Main Benefits

  • No manual cache clearing needed

  • Memory gets freed automatically

  • Zero memory leaks

  • Works with any size tensor

  • Thread-safe cache management

Get the colab

Thank You.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Advertise