Skip to main content
Back

Mistral launches its first ever Multimodal Model, Pixtral 12B

ACCESS THE MODEL IN HUGGINGFACE

September 11, 2024 6 min read

Hey,

Welcome to today's edition of AlphaSignal, a newsletter for developers by developers.

We identify and summarize the top 1% news, papers, models, and repos in the AI industry. 

IN TODAY'S SIGNAL

Read time: 6 min 27 sec

🎖️ Top News

📌 Ori

  • Kubernetes for AI/ML: Autoscaling, GPUs, quick deployment, simplified management. Pay-per-use pricing for optimizing costs.

⚡️ Trending Signals

📦 Top HF Models

  • Finegrain Box Segmenter: enable you to create pixel quality and high resolution cutouts for any object in a photo, by just naming the object.

  • RobustSAM: Segment Anything Robustly on Degraded Images.

  • Fish Speech: Multilingual text-to-speech model for instant voice cloning with low latency.


🧠 Python Tips

  • Bridge Design Pattern in Python to separate the high-level logic from the low-level operations.

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

Multimodal Model

Mistral Launches its first ever Multimodal Model, Pixtral 12B, with both language and vision processing capabilities

⇧ 4, 652 Likes

What's New

This release marks Mistral's entry into the competitive field of vision-language models, challenging established players like Meta.


Pixtral 12B Architecture and Specifications

  • The model builds upon Mistral's text-based Nemo 12B, incorporating a 400 million parameter vision adapter. 

  • It uses GeLU activation for the vision adapter, and 2D Rotary Position Embedding (RoPE) for the vision encoder. 

  • Pixtral 12B processes images up to 1024x1024 pixels, dividing them into 16x16 pixel patches.

  • 131,072 unique tokens in its vocabulary, allowing for nuanced language understanding and generation.


Key features include

  • Text backbone: Mistral Nemo 12B

  • Vision Adapter: 400M parameters

  • Vocabulary size: 131,072 tokens

  • Three new special tokens: 'img', 'img_break', 'img_end'

  • Model weights in bfloat16 format

  • Total download size: 24GB


Implementation and Access

  • Mistral has made the model weights available on the Hugging Face Hub. 

  • Developers can run the Pixtral 12B through the mistral_common python package, which now supports image input alongside text in user messages. 

  • The package can be installed via pip by running the following command

  • pip install --upgrade mistral_common

  • This integration allows for seamless incorporation of image data within the text processing pipeline, potentially enabling applications like visual question answering and image captioning.

  • And following Mistral's previous tradition, the model is distributed via peer-to-peer torrent network as well.


Performance and Benchmarks
As of the announcement, specific performance metrics and benchmarks for Pixtral 12B are not available. The model's capabilities in comparison to other multimodal models like GPT-4V or CLIP remain to be seen. Inference speed and resource requirements are also yet to be disclosed.


Future Developments and Applications


While Mistral has not provided specific guidelines for fine-tuning or adaptation, the release of Pixtral 12B opens up new possibilities. The model's large scale and advanced architecture could enable more sophisticated AI applications across various industries, from content creation to data analysis.


Access the model in huggingface

Build, deploy and scale on Ori Serverless Kubernetes

Leverage the power of Kubernetes, hassle-free management and robust scalability for your AI/ML workloads. Our pay-per-use pricing means you only pay for the resources you consume, helping you to optimize your GPU costs whilst scaling with user demand.  

  • Autoscaling: Adapt your AI infrastructure to user demand, while optimizing your costs. 

  • Powerful GPUs: Choose between our NVIDIA H100, L40S or L4 GPUs in either full or fractional configurations.

  • Get your models to market: No waiting for GPUs or approvals needed, create a cluster in less than a minute.

  • Kubernetes minus the complexity: Experience the benefits fo a full-scale control plane with a serverless implementation designed for MLOps.


Sign up to Ori and get started with $100 in free credit!

Start integrating today

partner with us

TRENDING SIGNALS

LLM Inferencing

SambaNova Launches The World's Fastest AI Platform, Llama 3.1 405B at 132 tokens/sec and 70B at 570 tokens/sec

⇧ 604 Likes

Cloud LLM

Anthropic added Workspaces to the Anthropic Console, it lets you manage multiple Claude deployments, set custom spend or rate limits, group API keys and offers many more flexibilities

⇧ 683 Likes

OpenAI

OpenAI's Strawberry is rumored to release its reasoning-focused next model within the next two weeks

⇧ 1,274 Likes

AI Agent Tutorial

LangChain releases its first course, on LangGraph, to teach building agents with LangGraph's graph-based workflows, for free

⇧ 376 Likes

LLM

Arcee.ai releases First distilled Llama 3.1, SuperNova which is a distilled reasoning Llama 3.1 70B & 8B, outperforms Meta Llama 3.1 70B instruct across benchmarks

⇧ 489 Likes

Transform speech into meaning

Redefine what’s possible with voice data—all on one seamless API that evolves ahead of the industry and handles the heavy lifting.

Start building free with $50 credit

TOP MODELS

Object Detection

Finegrain-box-segmenter

⇧250 Downloads

Helps you create high-resolution, pixel-perfect object cutouts in images. You provide a bounding box, and it generates a 1024x1024 alpha mask. It outperforms traditional background removal and SAM2 models, achieving 97.4% S-measure and 96.7% Dice score on e-commerce datasets. MIT license.

Object Segmentation

RobustSAM

⇧ 69 Downloads

RobustSAM helps you segment objects in degraded images. It builds on SAM's zero-shot capabilities while improving low-quality image performance. With minimal parameter increase, it outperforms SAM on various tasks. Trained on the new Robust-Seg dataset (688K image-mask pairs), it optimizes in 30 hours on 8 GPUs. RobustSAM enhances SAM-based tasks like dehazing and deblurring.

Text-to-Speech

Fish Speech

⇧ 189 Downloads

Fish Speech 1.4 helps you generate realistic speech in 8 languages. You can clone voices instantly with ultra-low latency. The model uses 700K hours of training data and weighs only ~1GB. It's open-source, with weights available on the Huggingface Hub.

PYTHON TIP

"Bridge Design" Pattern to separate the high-level logic from the low-level operations

The Bridge Design Pattern separates abstraction from implementation. Abstraction defines the high-level logic. Implementor declares methods for concrete implementations.


Use Bridge when you need to switch implementations at runtime or extend abstractions and implementations separately. It's useful in cross-platform applications or when working with multiple databases.

Benefits include improved code flexibility and reduced class explosion. It helps manage complex class hierarchies.

Below shows how to implement Bridge in Python. You'll see how client code interacts with the abstraction, regardless of the specific implementation.



from __future__ import annotations
from abc import ABC, abstractmethod


class Abstraction:
    def __init__(self, implementation: Implementation) -> None:
        self.implementation = implementation

    def operation(self) -> str:
        return (f"Abstraction: Base operation with:
"
                f"")


class ExtendedAbstraction(Abstraction):
    def operation(self) -> str:
        return (f"ExtendedAbstraction: Extended operation with:
"
                f"")


class Implementation(ABC):
    @abstractmethod
    def operation_implementation(self) -> str:
        pass

class ConcreteImplementationA(Implementation):
    def operation_implementation(self) -> str:
        return "ConcreteImplementationA: result on the platform A."


class ConcreteImplementationB(Implementation):
    def operation_implementation(self) -> str:
        return "ConcreteImplementationB: result on the platform B."


def client_code(abstraction: Abstraction) -> None:
    # ...

    print(abstraction.operation(), end="")

    # ...


if __name__ == "__main__":

    implementation = ConcreteImplementationA()
    abstraction = Abstraction(implementation)
    client_code(abstraction)

    print("
")

    implementation = ConcreteImplementationB()
    abstraction = ExtendedAbstraction(implementation)
    client_code(abstraction)


Explanation:  The code defines Abstraction and Implementation hierarchies. Abstraction holds an Implementation reference, calling operation_implementation().

ExtendedAbstraction overrides operation(). ConcreteImplementationA/B provide platform-specific implementations. Client code works with any Abstraction-Implementation pair, demonstrating the pattern's flexibility. This structure allows independent evolution of abstractions and implementations, enhancing maintainability and extensibility.

Run the code in colab

Thank You.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Advertise