Skip to main content
Back

Karpathy: Train GPT-2 in 90 minutes for $20

CHECK THE REPO

May 29, 2024 4 min read

Hey,

Welcome to today's edition of AlphaSignal. 


Whether you are a researcher, engineer, developer, or data scientist, our summaries are there to keep you up-to-date with the latest breakthroughs in AI. 


Let's get into it!


Lior

.

IN TODAY'S SIGNAL

Top News

Latitude

Top 5 Signals

mlcon

  • Register Free for mlcon 3 Today: Gain insights from 30+ AI experts.

Top Repos

  • PR-Agent: AI-Powered tool for automated Pull Request analysis, feedback, suggestions

  • Tarsier enhances LLM-web page interaction,

  • RT-DETR outperforms YOLOs in speed and accuracy on object detection

PyTorch Tip

  • Use PyTorch's 'bottleneck' to pinpoint and address performance bottlenecks.

Read Time: 4 min 42 sec

Enjoying this newsletter?
Please forward it to a friend or colleague. It helps us keep this content free.

TOP NEWS

Language Models

Karpathy releases instructions on how to train GPT-2 in 90 minutes and $20

⇧ 4728 Likes

What's New

Karpathy released a method to train GPT-2 models quickly and cost-effectively. Training a tiny GPT-2 (124M parameters) model takes about 90 minutes and $20 using an 8xA100 GPU. The 350M version requires 14 hours and around $200. Training the full 1.6B model takes one week and $2.5k.


llm.c: LLM Training in Pure C/CUDA
This release is part of Karpathy’s llm.c repository, which focuses on LLM training in simple, pure C/CUDA. There’s no need for large frameworks like PyTorch (245MB) or cPython (107MB).

Training GPT-2 on CPU (fp32) requires about 1,000 lines of code in train_gpt2.c, while GPU training (with CUDA kernels) uses about 3,000 lines in train_gpt2.cu. The code compiles and runs instantly, matching the PyTorch reference implementation and slightly exceeding its speed.


FineWeb Dataset
The training uses the FineWeb dataset, considered higher quality than the original WebText. FineWeb comprises simple English text with minimal math or code, enabling more efficient model capacity use and addressing diminishing returns seen in the original GPT-2’s 100B tokens.


Code Snippets
To start GPU training, follow these steps:

pip install -r requirements.txt 

python dev/data/tinyshakespeare.py 

python train_gpt2.py 

make train_gpt2cu 

./train_gpt2cu


Training Instructions
Karpathy provides detailed setup instructions. Key training parameters include:

  • Batch size: 64
  • Sequence length: 1024 tokens
  • Total batch size per update: 524,288 tokens These configurations achieve about 60% model flops utilization on the A100 GPU.

Key Innovations and Metrics
The core innovation is the use of llm.c, a C/CUDA implementation for efficient model training. The training process on an 8xA100 GPU achieves around 178K tokens per second, with initial training loss at 7.577, targeting a final loss of about 3.29.

Future Plans and Hardware Considerations
Karpathy plans to explore the RedPajama dataset, which might better represent WebText, and aims to improve llm.c’s core functionalities. There is also a focus on adding multi-node training support. Transitioning to H100 GPUs could further reduce training costs, though availability remains a challenge.

Important Specifications

  • 124M model: 90 minutes, $20
  • 350M model: 14 hours, $200
  • 1.6B model: 1 week, $2.5k

Check the repo

Deploy Your GPUs in Seconds: Readers Get 10% OFF

Accelerate your AI training, fine-tuning, and inference workloads with dedicated instances powered by NVIDIA’s H100 Tensor Core GPUs, high-performance networking, and enterprise-grade hardware designed for Machine Learning engineers.


Use Latitude.sh’s amazingly fast platform to deploy AI clusters in seconds. Their super intuitive dashboard and pre-installed AI software get you from zero to training with just a few clicks.


Get instant access to on-demand plans with no upfront commitment and deploy instances according to your specific needs: options ranging from 1x to 8x H100 GPUs are available today!


Alphsignal readers get 10% off during your first 3 months, just use the code G3OFF10

Get started

partner with us

TRENDING SIGNALS

Code Assistants

Mistral releases Codestral, their first code model, fluent in 80+ programming languages and outperforms LLama3

⇧ 1561 Likes

Voice

New techniques speeds up Whisper inference by 5x

⇧ 1029 Likes

Browser

Opera is integrating Google's Gemini AI into its browser

⇧ 921 Likes

Depth Estimation

Marigold, one of the best depth models is now available as a pipeline on HF with tutorial

⇧ 881 Likes

Web Development

Gemini can be used in Chrome Dev Console to understand errors and warnings

⇧ 1102 Likes

Last chance! Free registration for mlcon 3, the AI & ML developers conference

Join 13,000+ AI & ML developers. With topics such as bringing generative AI to production and optimizing your AI infrastructure.

  • Connect with like-minded professionals and expand your network.

  • Learn about real-world AI applications from 30+ top AI experts

Sign up ↗️

TOP REPOS

Coding Tools

PR-Agent

☆ 4471 Stars

PR-Agent is a collection of tools and features built around leveraging LLMs to assist with various tasks related to code review and development workflows in a collaborative software environment.

It provides capabilities like: 

  • automatically generating pull request (PR) descriptions, summaries, code walkthroughs, and labels.

  • it can also perform automated code reviews, provide adjustable feedback on potential issues, security concerns, and review effort estimations.

  • it offers code suggestions for improving PRs, answers free-text questions about the code changes, automatically updates changelogs, finds similar issues or code components, and generates documentation for changed methods, functions, and classes.

Web Agents

Tarsier

☆ 1161 Stars

Tarsier is a tool suite that solves the following problems regarding using LLMs for web interaction: 

  • Feeding the webpage to the LLM (HTML, Accessibility Tree, Screenshot)
  • Mapping LLM responses back to web elements
  • Informing a text-only LLM about the page's visual structure

Tarsier visually tags interactable elements on a page (buttons, links, or input fields) for an LLM to take actions upon. The repo also presents an OCR algorithm to convert a page screenshot into a whitespace-structured string (almost like ASCII art) that an LLM even without vision can understand.

Object Detection

RT-DETR

☆ 1481 Stars

This repository is the official implementation of the paper DETRs Beat YOLOs On Real-Time Object Detection.


It presents Real-Time DEtection TRansformer (RT-DETR, aka RTDETR), the first real-time end-to-end object detector, outperforming previously advanced YOLOs in both speed and accuracy.


The repository contains the code and the weights associated with the different versions of the RT-DETR model, that have between 20 an 76 million parameters.

PYTORCH TIP

Model Training

Using PyTorch's "bottleneck" module

The "torch.utils.bottleneck" module is a powerful tool that can help you identify performance bottlenecks in your PyTorch code, allowing you to optimize your models and training processes for improved efficiency.


This module provides different modes, such as 'cpu' for profiling CPU-bound operations and 'cuda' for profiling CUDA kernels, allowing you to pinpoint bottlenecks in different parts of your PyTorch pipeline.


After running the code snippet below, calling bottleneck.show_bottlenecks() will display a report highlighting the most time-consuming operations in your PyTorch code, along with their execution times, GPU utilization, and other relevant metrics.


Here's how you can use this module in an easy example:


import torch
from torch.utils.bottleneck import bottleneck

model = YourModelDefinition()
inputs = torch.randn(batch_size, input_shape)

with bottleneck(mode='full'):
    outputs = model(inputs)
    loss = loss_function(outputs)
    loss.backward()

bottleneck.show_bottlenecks()


Ghita is an editor at AlphaSignal and AI Engineer at Clear Ventures. Experienced in Computer Vision and Language models, she holds 2 masters in Applied Mathematics.

Thank You.

Want to promote your company, product, job, or event to 180,000+ AI researchers and engineers?

Sponsor