Hey, | Welcome to AlphaSignal, your weekly guide to the latest and most promising developments in the world of AI.
In this edition, we've meticulously scoured GitHub, HuggingFace, and every relevant code repository to identify the most trending and groundbreaking projects.
Let's get into it!
Lior |
|
|
|
IN TODAY'S EMAIL | Repo Highlight: LLMs-from-scratch Trending Repos: Depth-Anything, moondream Trending Models/Spaces Pytorch Tip: mixed precision training Python Tip: contextlib
| Reading time: 4 min 43 sec |
|
|
|
HIGHLIGHT REPO | Build a Large Language Model (From Scratch) |  | What's New | "Build a Large Language Model (From Scratch)" by Sebastian Raschka blew up on Github this week and collected over 5000 stars. These resources are designed to provide hands-on experience and foundational knowledge necessary for building LLMs.
They focus on essential aspects such as data handling, attention mechanisms, and framework familiarity, setting the stage for more advanced topics in upcoming chapters.
Available content includes:
Chapter 2: Working with Text Data - Jupyter notebooks (ch02.ipynb, dataloader.ipynb) focusing on handling and processing text data.
- Supplementary materials in the ./ch02 folder for in-depth exploration.
Chapter 3: Coding Attention Mechanisms - Notebooks (ch03.ipynb, multihead-attention.ipynb) dedicated to the implementation of attention mechanisms, crucial for LLMs.
- All code and additional resources located in the ./ch03 folder.
Appendix A: Introduction to PyTorch - Several notebooks (code-part1.ipynb, code-part2.ipynb, DDP-script.py) introducing PyTorch, a key framework for LLM development.
- Exercise solutions and further guidance for Python and package installations in the ./appendix-A folder.
| | View the repo |
|
|
|
 | Unlock the Future of AI: Master RAG Models @ Gretel's Expert Workshop!
Join the cutting edge of AI technology with Gretel's exclusive virtual workshop on Fast Tracking RAG Model Evaluation using Synthetic Data. Scheduled for February 5, 2024, this event is a must-attend for AI enthusiasts and professionals. Learn from industry experts like Alex Watson from Gretel and Sam Kemp from Microsoft as they explore the revolutionary impact of synthetic data on RAG techniques and data privacy.
Discover how Gretel's Tabular LLM model can generate diverse datasets, enhancing RAG system robustness through varied, realistic Q&A simulations. Don't miss this opportunity to advance your knowledge and skills in generative AI and large language models.
Register now for an insightful session with a live Q&A! | | Secure your spot | partner with us | |
| |
|
|
TOP OF GITHUB | OLMo (☆ 167) Arguably the biggest development in open-source LLMs yet. OLMo, a state-of-the-art open language model by AI2, offers training and experimentation frameworks on Hugging Face and GitHub. It includes variants at 7B and 1B scales, trained on 2T+ tokens, with full training data, model weights, and evaluation code. | | Depth-Anything (☆ 3.9k) Tik Tok just released Depth Anything, a powerful depth estimation foundation model that can deal with any images under any circumstances.
Depth Anything is trained on 1.5M labeled images and 62M+ unlabeled images jointly, it improves zero-shot and in-domain depth prediction, refines ControlNet, and transfers to semantic segmentation. | | RPG-DiffusionMaster (☆ 1k) The RPG framework enables text-to-image generation using multimodal LLMs, supporting high-resolution outputs and customizable regional diffusion. It integrates base diffusion models like SD v1.4-v2.1 and supports local LLMs with over 13 billion parameters, optimizing for computational efficiency and image quality. | | stanfordnlp / dspy (☆ 5.5k) DSPy framework enables the optimization of LM prompts and weights, streamlining the integration of LMs in complex systems. It systematically adjusts program parameters, allowing LMs like GPT-3.5 or GPT-4 to perform tasks more reliably by reducing failure rates and enhancing quality. | | LLMLingua (☆ 2.6k) LLMLingua and LongLLMLingua compress prompts for LLMs like GPT-4, reducing token count by up to 20x, enabling efficient inference and longer context processing with minimal performance loss, and integrating seamlessly into LlamaIndex for enhanced RAG performance. | |
| |
|
|
PYTORCH TIP | mixed precision training | Use torch.cuda.amp in PyTorch for mixed precision training. This method mixes 32-bit and 16-bit data to reduce memory use and speed up model training, without much loss in accuracy.
It takes advantage of the quick computing of 16-bit data and controls precision by handling specific operations in 32-bit. This approach offers a balance between speed and accuracy in training models. | import torch from torch.cuda.amp import autocast, GradScaler from torch import nn, optim
model = nn.Linear(10, 2).cuda() optimizer = optim.SGD(model.parameters(), lr=0.01) scaler = GradScaler()
data, target = torch.randn(5, 10).cuda(), torch.randint(0, 2, (5,)).cuda()
optimizer.zero_grad()
with autocast(): loss = nn.functional.cross_entropy(model(data), target)
scaler.scale(loss).backward() scaler.step(optimizer) scaler.update()
|
|
|
|
TOP MODEL/SPACES | CodeLlama-70b-Instruct-hf Code Llama offers pretrained generative text models for code synthesis and understanding, ranging from 7B to 70B parameters. It supports code completion, infilling, and chat interactions, optimizing for ML development tasks. | | moondream-1 Rival to GPT-4 Vision, moondream1 is a tiny 1.6B parameter vision language model that punches above its weight. Upload any pictures. | |
| |
|
|
PYTHON TIP | contextlib | Use Python's contextlib module to create a timer that can be used as a context manager. This is particularly useful for tracking performance of different parts of your code. With this approach, you can easily measure the execution time of a block of code without cluttering your code with timing logic.
This code defines a timer context manager. When you wrap a block of code with this timer, it prints out how long the block took to run. This is very handy for performance profiling in any Python code, especially for measuring model training and inference times. | from contextlib import contextmanager import time
@contextmanager deftimer(label): start = time.time() try: yield finally: end = time.time() print(f': seconds')
# Usage: with timer('Model training'): # Your code block here, e.g.: # train your model
|
|
|
|
Want to promote your company, product, job, or event to 150,000+ AI researchers and engineers? You can reach out here. |
|
|
|
|