Skip to main content
Back

Nvidia's Big Move

TRY CHAT WITH RTX

February 14, 2024 4 min read

Hey,

Welcome to AlphaSignal, your weekly guide to the latest and most promising developments in the world of AI.


Let's get into it,


Lior

IN TODAY'S EMAIL

  • NVIDIA Launches RTX

  • Top Announcements

  • Tutorial: How to Fine-Tune an LLM

Reading time: 4 min 17 sec

HIGHLIGHT REPO

NVIDIA Launches RTX - A Personalized LLM Chatbot

What's New

Nvidia just released "Chat with RTX" a local app that allows you to create a personal AI chatbot (LLM) based on your own content. 


Rather than searching through notes or saved content, users can simply type queries. For example, one could ask, “What was the restaurant my partner recommended while in Las Vegas?” and Chat with RTX will scan local files the user points it to and provide the answer with context.


Key Features:

  • Retrieval-Augmented Generation (RAG): Uses Mistral or Llama 2 LLMs for text generation, accelerated by TensorRT-LLM for faster response times.
  • Local Processing: Runs entirely on an RTX 30/40 GPU (≥8GB VRAM) and Windows PC, avoiding cloud latency and privacy concerns.

  • Data Ingestion: Supports various formats like txt, pdf, docx, and xml, alongside YouTube video transcripts

Potential Applications:

  • Data research: Analyze personal documents, transcripts, or videos for specific information with near-instant results.

  • Content summarization: Quickly grasp key points from YouTube videos or lengthy documents.

  • Fact-checking: Verify data points within PDFs or other sources.

Current Limitations:

  • Early Demo: Known issues include inaccurate source attribution, context-free responses, and crashes with large datasets.

  • Resource Intensive: Installation requires 40GB disk space and consumes ~3GB RAM, potentially impacting system performance.
Try chat with rtx

Don't Miss The Best Web Conference: AI in Production


Large Language Models have taken the world by storm. But what are the real use cases? What are the challenges in productionizing them?

In this event, you will hear from practitioners about how they are dealing with things such as cost optimization, latency requirements, trust of output and debugging.

  • 70+ speakers from tech companies like Netflix, Uber, Doordash, Microsoft, Google, LinkedIn, Databricks and also top AI startups like Perplexity AI, HuggingFace and Cohere.

  • The event is 100% virtual taking place on two thursdays feb 15th and 22nd

  • ⁠3 tracks: one for engineering, one for AI product folks and one workshop track

  • ⁠The breaks will be full of live musical improv, games and guided meditations

  • The best part? It's completely free.

Register now

partner with us

TOP NEWS

OpenAI Adds Memory to ChatGPT

---------------------

OpenAI's latest ChatGPT update introduces memory across chats, enhancing personalization. Users can manage, delete, or disable this feature, ensuring privacy. Initially limited to select users, a wider rollout is planned, aiming to improve user experience with controlled data handling.

Cohere Announces Aya, A Massive MultiLingual LLM

---------------------

Cohere For AI launched Aya, an open-source, multilingual LLM covering 101 languages, with a 513M-item dataset. Developed by 3,000 researchers,  Aya is one of the largest open science endeavors in machine learning to date. The result is a fully open-sourced dataset and model.

Apple Releases ‘MGIE’, an Instruction-Based Image Editing Model

---------------------

Apple unveiled MGIE, an open-source AI for instruction-based image editing using multimodal large language models (MLLMs) for natural language processing and pixel-level manipulation. MGIE allows Photoshop-style modifications and both global and local photo optimizations through expressive, instruction-derived edits.

Dive Into The Future Of Driving And Master Computer Vision In Autonomous Vehicles

Unlock secrets to enhance safety, reliability, and real-time decision-making. Discover rigorous testing methodologies for unmatched system performance. Tackle the unique challenges of vision technology in self-driving cars.


Reserve your spot in our exclusive webinar today – don’t miss out!


Register Now →

TUTORIAL

How to Fine-Tune an LLM more efficiently with QLoRA

Parameter-efficient training (PEFT) techniques offer a way to fine-tune large language models (LLMs) on custom datasets with minimal computational resources. Those techniques only update a small number of parameters during the fine-tuning process, and freeze the rest. 


LoRA is a PET technique that fine-tunes two smaller matrices that approximate the model. Its Quantized version, QLoRA, uses a smaller precision on those two proxy matrices.


Here's a broad overview of how to use QLoRA to fine-tune an LLM. 


1. Install Required Libraries

  • Key Libraries:

    • bitsandbytes for efficient model loading and quantization.

    • transformers for pre-trained models and utilities.

    • peft for parameter-efficient fine-tuning.

2. Prepare the Model for QLoRA

  • Configure the model to use 4-bit quantization using BitsAndBytesConfig.

  • Load a pre-trained model in 4-bit format using the transformers library and the 4-bit config.

5. Tokenization and Pre-processing

  • Configure the tokenizer with necessary settings for optimal memory usage.

  • Pre-process the dataset to match the model's expected input format (completion, summarization, sentiment analysis, etc). 

  • Tokenize the pre-processed dataset, batch and shuffle it.

6. Fine-Tuning with QLoRA

  • Prepare the model for QLoRA using the prepare_model_for_kbit_training() function of the peft library .

  • Define a LoraConfig for fine-tuning, specifying ranks, modules, and other parameters using the peft library, and update the model using this config.

  • Use transformers.Trainer with custom training arguments to fine-tune the adapter.

Code Snippet for Fine-Tuning Setup:


from transformers import (TrainingArguments, Trainer)

from peft import (LoraConfig, get_peft_model
                  prepare_model_for_kbit_training)

# Prep QLoRA model
model = prepare_model_for_kbit_training(
    pretrained_model)

# Set LoRA config
lora_config = LoraConfig(r=32, lora_alpha=32,
    target_modules=['q_proj', 'k_proj',
        'v_proj', 'dense'])

# Get PEFT model
peft_model = get_peft_model(
    model, lora_config)

# Training args
training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=3,
    per_device_train_batch_size=4,
    logging_dir='./logs')

# Init Trainer
trainer = Trainer(
    model=peft_model, args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset)

# Fine-tune
trainer.train()

Thank You

Want to promote your company, product, job, or event to 170,000+ AI researchers and engineers? You can reach out here.