Skip to main content
Back

IBM Granite 3.0 8B Surpasses Llama 3.1 on OpenLLM Benchmark

READ MORE

October 23, 2024 6 min read

Hey,

Welcome to today's edition of AlphaSignal, a newsletter for developers by developers.

We identify and summarize the top 1% news, papers, models, and repos in the AI industry. 

IN TODAY'S SIGNAL

Read time: 6 min 9 sec

🎖️ Top News


📌 Ori

⚡️ Trending Signals


📌 Lambda


📦 Top HF Models

🧠 Pytorch Tip

  • Custom autograd functions: build your own differentiable Pytorch operations

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

IBM

IBM Releases Granite 3.0: New Open-Source LLM Family Optimized for Enterprise Scale

⇧ 3,486 Likes

What's New

IBM's Granite 3.0 introduces multiple language models ranging from 2B to 8B parameters, released under Apache 2.0 license. The models support 12 natural languages and 116 programming languages, trained on 12 trillion tokens using a novel two-stage approach.


Key Highlights

  • Granite 3.0 8B scores 37.6 on OpenLLM Leaderboard, surpassing Llama 3.1 8B (37.3)

  • Achieves 3-23x cost reduction vs larger models in production tests

  • Delivers 2x latency improvement through speculative decoding

  • Processes enterprise tasks with <1B parameters using MoE architecture

  • Supports 128K context window (upcoming)

Architecture and Technical Implementation

  • Granite 3.0 8B achieves 37.6 score on OpenLLM Leaderboard, outperforming Llama 3.1 8B (37.3) and similar-sized Mistral/Meta models on RAG benchmarks.

  • MoE variants (3B-A800M) maintain performance while running with <1B parameters at inference time, making them viable for CPU deployments.

  • Speculative decoding implementation delivers 2x improvement in both latency and throughput for production workloads


Architecture and Technical Implementation

The core innovation lies in the Mixture-of-Experts (MoE) architecture, enabling CPU-based deployments while maintaining performance. The 3B-A800M and 1B-A400M variants run efficiently with less than 1B parameters at inference time. The models implement speculative decoding for reduced latency and enhanced throughput in production environments.


Training and Optimization

The two-stage training method optimizes data quality through thousands of experiments. The InstructLab alignment technique enables fine-tuning with enterprise data, particularly effective for RAG, classification, and code generation tasks. The training architecture processes multiple programming languages through specialized token optimization.


Deployment and Integration

The models deploy across multiple platforms including NVIDIA NIM, Google Cloud Vertex AI, and Hugging Face. Enterprise deployments benefit from IP indemnification through watsonx platform. The architecture supports both edge computing and CPU-based implementations, with specific optimizations for low-latency applications.


Technical Capabilities

The models excel in enterprise-specific tasks like Retrieval Augmented Generation, tool use, and advanced reasoning. The architecture supports multi-modal document understanding and implements efficient token processing for extended context windows.

Read more

Flexible AI infrastructure to build anything imaginable

Ori Private Cloud now delivers isolated GPU infrastructure for specialized AI workloads at scale.


  • Zero shared resources: Single-tenant infrastructure with complete access control.

  • Full GPU flexibility: Scalable clusters that adapt to specific compute needs.

  • Global compliance ready: Built-in support for regulatory requirements and certifications.

  • Expert HPC support: Direct access to HPC specialists throughout project lifecycle.

‍‍‍‌‍Contact the team for a free consultation and to set up a private GPU cloud deployment.

Learn more

partner with us

TRENDING SIGNALS

Perplexity

Perplexity introduces Reasoning Mode to enhance multi-layered query processing for complex AI research

⇧ 4,370 Likes

ElevenLabs 

ElevenLabs introduces text-to-voice generation system, enabling us to create custom voices instantly

⇧ 1,600 Likes

Meta

Meta unveils self-contained LLM training framework Lingua for rapid research experimentation in PyTorch

⇧ 1,000 Likes

Video Generation

Genmo releases Mochi 1, setting new SOTA benchmarks in open-source video generation

⇧ 5,551 Likes

Google

Google publishes about how they uses probabilistic reasoning framework to enhance LLM inference accuracy

⇧ 305 Likes

New On-Demand 1x, 2x, and 4x NVIDIA H100 SXM GPU Instances

Lambda just added new instances in their public cloud that deliver up to 51% higher performance than PCIe models.


AI developers now have cheaper, flexible access to powerful compute without needing full-node 8x setups.

Spin up an instance

TOP MODELS

Text to Video

Allegro

⇧49 Downloads

Helps you create 6-second, 720x1280 videos from text descriptions at 15 FPS. The open-source model combines a 175M VideoVAE and 2.8B VideoDiT to generate diverse scenes, from close-ups to dynamic content.

Text to Image

stable-diffusion-3.5-large

⇧ 10,,991 Downloads

Stable Diffusion 3.5 helps you generate customizable AI images with better prompt accuracy. The 8B parameter Large model delivers highest quality, while Large Turbo offers faster generation. The 2.6B Medium model runs on edge devices.


Key technical updates

  • MMDiT-X architecture combines diffusion and transformer techniques for multi-resolution generation.

  • Query-Key Normalization makes fine-tuning more stable.

  • Free for non-commercial use and commercial use under $1M revenue.

  • Available via API and Hugging Face.

  • ControlNets coming soon for spatial control and color preservation.

LLM

Llama-3.1-Nemotron-70B-Instruct-HF

⇧ 89,133 Downloads

You can now run Meta's largest language model directly from Hugging Face. This 70B parameter model outperforms GPT-4-Turbo and Claude 3.5 on key benchmarks - scoring 85.0 on Arena Hard, 57.6 on AlpacaEval 2 LC, and 8.98 on MT-Bench.


Key Highlights

  • Runs on 2+ 80GB GPUs with 150GB disk space.

  • Handles 128k input tokens, outputs 4k tokens max.

  • Built on Llama 3.1, fine-tuned with REINFORCE using HelpSteer2 dataset.

  • Works with standard HF transformers code, no special prompting needed.

  • Free hosted inference available through NVIDIA's API

PYTORCH TIP

Custom autograd functions: build your own differentiable Pytorch operations

An Autograd Function in PyTorch is a class that defines how to compute custom forward and backward passes for a new operation, letting you specify your own gradient computation rules. It's the building block that lets you create new differentiable operations beyond PyTorch's built-in ones.

You can define custom backward pass operations in PyTorch when the built-in autograd isn't sufficient. This is useful for complex gradient computations, custom optimization algorithms, or specialized layer operations.


Here's a simple example that implements a custom ReLU with a tweaked gradient:



import torch
from torch.autograd import Function

class CustomReLU(Function):
    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input < 0] = 0.1
        # Modified gradient for negative values
        return grad_input

# Usage
x = torch.randn(5, requires_grad=True)
custom_relu = CustomReLU.apply
output = custom_relu(x)
output.sum().backward()
print("Input:", x)
print("Gradient:", x.grad)


This code creates a ReLU variant where negative inputs get a small positive gradient (0.1) instead of zero. This helps prevent "dying ReLU" problems in deep networks by allowing some gradient flow even for negative inputs.

Get the colab

Thank You.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Advertise