Skip to main content
Back

Implementing LLaMa 3 From Scratch

CHECK THE REPO

May 21, 2024 5 min read

Hey,

Welcome to today's edition of AlphaSignal. 


Whether you are a researcher, engineer, developer, or data scientist, our summaries are there to keep you up-to-date with the latest breakthroughs in AI. 


Let's get into it!


Lior

.

IN TODAY'S SIGNAL

Top Repo

MLCON 3

Top Signals

Paragon

Top Papers

Python Tip

  • Use contextlib to create custom context managers for efficient resource management.

Read Time: 5 min 41 sec

Enjoying this newsletter?
Please forward it to a friend or colleague. It helps us keep this content free.

TOP REPO

Language Models

LLaMa 3 Implemented From Scratch in Python

⇧ 4720 Likes

What's New

A new repository, praised by Andrej Karpathy, provides a detailed implementation of Llama3, breaking down each component from matrix multiplication in attention mechanisms to positional encoding. Users can load tensors directly from Meta's official Llama3 model file after downloading the weights.

Core Components Covered

  • Tokenization Process: Uses the tiktoken library for tokenization with Meta-Llama-3-8B's tokenizer model. Special tokens and mergeable ranks are loaded, converting text into tokens and embeddings after RMS normalization.

  • Attention Mechanism: Details weights for queries, keys, values, and outputs. Explains splitting query vectors into pairs, rotating them using RoPE (rotary positional embedding), and obtaining complex numbers for each token's query element. Uses the rope_theta parameter from the model's config, resulting in a rotated query vector.

  • Multi-Head Attention Operations: Includes matrix multiplication for query-key scores and masking future tokens during training. The attention scores matrix maps token relationships. Values are computed similarly, resulting in a final attention vector.

  • Feedforward Network: Uses SwiGLU to process edited embeddings further. Each layer performs these operations, with the final embedding normalized and decoded into token predictions after 32 transformer layers.

  • Visualizations and Practical Examples: Offers practical examples and visualizations, such as heatmaps for attention scores, to aid in understanding Llama3's architecture.

This detailed, step-by-step implementation makes Llama3's architecture and functioning accessible for further experimentation and research.

Check the repo

Last chance! Free registration for MLCON 3, the AI & ML Developers Conference

Join a community of 13k+ AI & ML developers from Intel, NASA, Spotify, Walmart, Pinterest, Lenovo, and dozens more in a series of virtual and in-person sessions. With topics such as bringing generative AI to production and optimizing your AI infrastructure.


Sign up Today to:

  • Learn from top AI leaders in the industry.

  • Connect with like-minded professionals and expand your network.

  • Experience the thrill of cutting-edge AI technology firsthand.

  • Learn about real-world AI applications from 30+ top AI experts.

Apply now

partner with us

TRENDING SIGNALS

Interview

AI pioneer, Geoffrey Hinton, discusses Ilya's theory and the need for UBI in a new 45-min interview

⇧ 1203 Likes

Language Models

New open-source model Smaug Llama 3 70B is on par with GPT-4 Turbo on MT-Bench

⇧ 981 Likes

Classification Models

New classification algorithm outperforms CatBoost, XGBoost, LightGBM on 5 benchmarks, accuracy, and response time

⇧ 318 Likes

Coding Assistants

Cursor rolls out near-instant code edits model outperforming GPT-4

⇧ 551 Likes

Audio

ElevenLabs introduces Audio Native, an embeddable audio player that automatically narrates your blog or news site

⇧ 826 Likes

Easily Ingest External Data for RAG/fine-tuning

Paragon allows users to build useful AI SaaS products by easily extracting users’ data from 100+ sources and adding it to their multi-tenant RAG pipeline.


Leading AI SaaS companies like copy. ai, Writesonic, Tavus and Pryon use Paragon’s SDK and embedded platform as their ingestion engine to pull in customer-specific, contextual data for their RAG pipeline.

TRY NOW ↗️

TOP PAPERS

Text-to-image

Chameleon: Mixed-Modal Early-Fusion Foundation Models

⇧ 2021

Problem: Most multimodal models available process different modalities separately, which limits their ability to integrate across modalities and generate documents with arbitrary sequences of images and text.


Solution: Chameleon is a family of early-fusion, token-based mixed-modal models that can understand and generate images and text in any sequence. Its fully token-based architecture allows for seamless information integration across modalities. By quantizing images into discrete tokens and training on mixed-modal data from scratch, Chameleon learns to jointly reason over image and text in a brand new way.


Results: Chameleon achieves state-of-the-art performance on image captioning and visual QA benchmarks, is competitive with large language models on text-only tasks and shows a strong performance on new mixed-modal open-ended QA evaluation. 

3D Generation

CAT3D: Create Anything in 3D with Multi-View Diffusion Models

⇧ 1249

Problem: Existing methods for 3D reconstruction from limited views or a single image often produce low-quality results or are computationally expensive.


Solution: A multi-view diffusion model is trained to generate multiple output views given one or more input views and camera poses. It uses 3D self-attention and "raymap" pose embeddings. A robust 3D reconstruction pipeline with perceptual losses is applied to the generated views to produce a 3D representation.


Results: CAT3D outperforms prior work on few-view and single image 3D reconstruction benchmarks, both quantitatively and qualitatively. It can generate high-quality 3D content in as little as one minute, an order of magnitude faster than previous methods.

Language Models

LoRA Learns Less and Forgets Less

⇧ 1050

Problem: Low-Rank Adaptation (LoRA) is a parameter-efficient finetuning method for large language models, but its performance compared to full finetuning is unclear, particularly in specialized domains like programming and mathematics.


Solution: The study compares LoRA and full finetuning on programming and mathematics, using instruction finetuning (≈100K prompt-response pairs) and continued pretraining (≈10B tokens). It evaluates performance and regularization effects, analyzing perturbation ranks and comparing with techniques like weight decay and dropout.


Results: LoRA underperforms full finetuning in target domains but better preserves base model performance. LoRA achieves a rank 10-100X lower than full finetuning, explaining performance gaps. Applying LoRA to all layers yields better results, and it offers stronger regularization, reducing overfitting more effectively than dropout and weight decay.

PYTHON TIP

Resource Management

Create Custom Context Managers Using contextlib

Python's contextlib module provides utilities for working with the with statement, which is used for resource management. The with statement ensures that resources are properly acquired and released, even in the presence of exceptions.


The contextlib module offers several classes and functions to create context managers, which are objects that define the __enter__ and __exit__ methods. The __enter__ method is called when entering the with block, and the __exit__ method is called when exiting, regardless of whether an exception occurred or not.

Context managers can be used for a variety of tasks, including:

  1. File operations: Automatically closing files after use.
  2. Thread and process locking: Acquiring and releasing locks.
  3. Database connections: Connecting to and disconnecting from databases.
  4. Temporary environments: Creating and cleaning up temporary environments.
  5. Logging: Temporarily changing logging levels.
  6. Timing operations: Measuring the execution time of a block of code.

Here's a quick example of how you can create a custom context manager using contextlib to time a block of code:


import time
from contextlib import contextmanager

@contextmanager
def timed_block(label):
    start = time.time()
    try:
        yield
    finally:
        end = time.time()
        print(f": seconds")

# Usage:
with timed_block("quick computation"):
    # Some quick computations
    sum(range(1000000))


Ghita is an editor at AlphaSignal and AI Engineer at Clear Ventures. Experienced in Computer Vision and Language models, she holds 2 masters in Applied Mathematics.

Thank You.

Want to promote your company, product, job, or event to 180,000+ AI researchers and engineers?

Sponsor