Skip to main content
Back

Google's Secret Model Outperforms Everyone

TRY GEMINI 0801

August 5, 2024 5 min read

Hey,

Welcome to today's edition of AlphaSignal, a newsletter for developers by developers.

We identify and summarize the top 1% news, papers, models, and repos in the AI industry. 

IN TODAY'S SIGNAL

Read time: 5 min 1 sec

🎖️ Top News

📌 Galileo

⚡️ Trending Signals

📄 Top 3 Papers

🧠 Python Tip

  • How to perform fast vector similarity search in Python with sqlite-vec.

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

Language Models

Deepmind secretly drops the most powerful LLM in the industry

⇧ 1532 Likes

What's New

DeepMind has launched Gemini 1.5 Pro (Experimental 0801), a large language model that tops the LMSYS leaderboard. It scored 1300, surpassing models like GPT-4.0 with 1286 and Claude 3.5 Sonnet with 1271. This model has gained strong community approval, reflected in its 12,672 votes.


Key Innovations and Architecture

Gemini 1.5 Pro uses a Mixture-of-Experts (MoE) architecture that efficiently distributes tasks across specialized subnetworks, enabling the model to handle complex tasks more effectively. Its context window of 2 million tokens allows it to process extensive information, far surpassing GPT-4's 32,000-token capacity.


Multilingual and Technical Proficiency

This model excels in multilingual tasks, ranking first in languages such as Chinese, Japanese, German, and Russian. In technical domains, it shows strong performance:

  • Math: Ranks 1st to 3rd
  • Instruction-Following: Ranks 1st to 2nd
  • Coding: Ranks 3rd to 5th
  • Hard Prompts (English): Ranks 2nd to 5th


Integration and Multimodal Capabilities

You can test Gemini 1.5 Pro through Google AI Studio and the Gemini API. It integrates easily into applications via platforms like AI Studio and Vertex AI. Its multimodal capabilities allow it to process text, images, and video, making it adaptable for various data types and applications.

Try gemini 0801

LLM Hallucination Index: RAG Special

Ever wondered what's the best LLM to use for RAG? The LLM Hallucination Index can be used to answer this.

Galileo evaluated 22 open/closed models across 3 RAG tasks and provides insights on cost, verbosity, and performance to help choose the optimal LLM depending on the use case.

Some key findings:
1. Anthropic outperforms OpenAI – Claude beats GPT.
2. Open-source is quickly closing the gap with closed-source.
3. Larger is not always better – Llama outperformed bigger models.

Read the report

partner with us

TRENDING SIGNALS

Agents

LangChain releases LangGraph Studio: an agent IDE to develop LLM applications that enables visualization, interaction, and debugging

⇧ 1829 Likes

Image Generation

Former Stable Diffusion developers launch Flux, an open-source AI Image Generator that beats Midjourney and SD3

⇧ 1225 Likes

Prompt Engineering

Character.AI open-sources Prompt Poet: a tool for effortless and precise AI prompt creation

⇧ 451 Likes

Models

GitHub launches an integrated model playground for AI developers

⇧ 629 Likes

Image-to-3D

Stability announces Stable Fast 3D: a model that transforms a single input image into a detailed 3D asset in just 0.5 sec

⇧ 2311 Likes

TOP PAPERS

Model Training

Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget

⇧ 2821 Likes

Problem

Training large-scale text-to-image diffusion models requires significant computational resources, creating barriers for researchers with limited budgets. Current models need proprietary datasets or large image collections and incur costs as high as $28,400, making participation difficult.


Solution

The authors propose a deferred masking strategy that preprocesses image patches before masking, allowing a 75% reduction in computational cost without significant performance loss. This approach enables training a 1.16 billion parameter sparse transformer on a micro-budget using only 37 million images.


Results

The model achieves a Fréchet Inception Distance (FID) of 12.7 on COCO, reducing training costs by 118× compared to Stable Diffusion, with a $1,890 cost in 2.6 days on 8×H100 GPUs. 

Foundational Models

Apple Intelligence Foundation Language Models

⇧ 908 Likes

Problem

Efficient and privacy-focused language models were needed for real-time AI applications on devices, ensuring user privacy without sacrificing performance. Existing models often struggled with speed, efficiency, and privacy compliance, especially for on-device use.


Solution

Apple introduced two models: AFM-on-device (∼3 billion parameters) for efficient processing on devices, and AFM-server for Private Cloud Compute. These models leverage quantization and LoRA adapters to improve speed and efficiency while adhering to strict privacy standards by excluding personal data.


Results

AFM-server scored 75.3% on MMLU 5-shot and 72.4% on GSM8K, surpassing many existing benchmarks. AFM-on-device excelled in summarization tasks, demonstrating superior quality with high "good" response ratios.

Transformers

Mixture of A Million Experts

⇧ 1873 Likes

Problem

Traditional transformer models face increasing computational costs due to the linear growth of feedforward (FFW) layers as model size increases, limiting scalability and efficiency.


Solution

The "Mixture of A Million Experts" paper from DeepMind introduces the Parameter Efficient Expert Retrieval (PEER) architecture. PEER leverages product key retrieval for efficient routing among over a million tiny experts, decoupling computational cost from parameter count.


Results

In language modeling tasks, PEER outperformed dense FFW, coarse-grained MoE, and PKM layers. It achieved lower perplexities (e.g., 16.34 on C4 dataset) with reduced FLOP budgets (6e18 and 2e19), demonstrating superior efficiency and scalability.

PYTHON TIP

Using sqlite-vec for Vector Similarity Search in Python

sqlite-vec is a small SQLite extension for fast vector searches, written in pure C, with no dependencies. It allows storing and querying vectors in SQLite and runs on various platforms like Linux, MacOS, Windows, and more.


When to Use It: Use sqlite-vec when you need a lightweight, embedded solution for vector similarity search. It is ideal for applications that require vector data handling on-device or at the edge, such as mobile apps, IoT devices, or local AI model inference.

Applications:

  • Recommendation systems based on vector similarity
  • Image and audio search by embedding vectors
  • Real-time search in resource-constrained environments

Check the blog


pip install sqlite-vec

import sqlite3
import sqlite_vec
from sqlite_vec import serialize_float32

db = sqlite3.connect(":memory:")
db.enable_load_extension(True)

sqlite_vec.load(db)
db.enable_load_extension(False)

embedding = [0.1, 0.2, 0.3, 0.4]
result = db.execute('SELECT vec_length(?)',
        [serialize_float32(embedding)])

print(result.fetchone()[0])  # Outputs: 4

Note: For NumPy arrays, cast to 32-bit floats with .astype(np.float32).


Ensure SQLite version is 3.41 or higher for full feature compatibility. Use pysqlite3 if you need a newer SQLite version than your Python installation provides.

Thank You.

Want to promote your company, product, service, or event to 200,000+ AI researchers and engineers?

Sponsor