Skip to main content
Back

Meet the World's First "AI Scientist"

READ MORE

August 14, 2024 5 min read

Hey,

Welcome to today's edition of AlphaSignal, a newsletter for developers by developers.

We identify and summarize the top 1% news, papers, models, and repos in the AI industry. 

IN TODAY'S SIGNAL

Read time: 5 min 57 sec

🎖️ Top News

📌 Work OS

  • Start selling to Enterprises with just a few lines of code. Implement features like single sign-on in minutes instead of months.

⚡️ Trending Signals

📌 AssemnlyAI

  • AssemblyAI's API offers advanced speech AI, integrating speech-to-text, comprehension, and Claude 3 models for complete voice analysis.


🧠 Top Papers

🧠 Tutorial

  • How to Quantize Llama 3.1 using AutoRound library with Python.

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

AI Scientist

Sakana AI Introduces world’s first fully automatic research paper generation AI

⇧ 8774 Likes

What's New

Sakana AI revealed "The AI Scientist", an AI system that autonomously conducts the entire lifecyle of scientific research, using foundational LLMs like GPT4, Sonnet 3.5, DeepSeek Code. Average computing costs of producing a full paper is less than $15. 


The papers produced by "The AI Scientist," exceed the acceptance threshold at a top machine learning conference as judged by their automated reviewer. 

The github code for 'AI-Scientist'' is available under Apache-2.0 license and they also publised a detailed paper.


Complete Automation of the Research life-cycle

The AI Scientist handles all stages of research, including generating ideas,  research directions, coding, executing experiments, analyzing results, and performing peer reviews.


It mimics human researchers by conducting literature searches, planning experiments, generating figures, and reviewing manuscripts. 


Demonstrated Capabilities

The AI Scientist has been applied to machine learning research, specifically in diffusion models, transformer-based language models, producing papers with novel contributions. The researchers have released many full papers fully produced by "AI Scientist"

Operational Process

- "The AI Scientist" begins with a broad research direction and an existing codebase, such as open-source code from GitHub.
- Independently executes the discovery process without human intervention.
- Implement an automated peer review process to evaluate generated papers, write feedback, and further improve results.

- Overall workflow uses chain-of-thought prompting and self-reflection to improve LLM reasoning for research tasks and incorporates web search to check idea novelty.

Current Limitations

The AI Scientist lacks computer vision capabilities, restricting its ability to address visual issues in papers. It can produce inaccurate results due to AI hallucinations and make critical errors in data evaluation, such as comparing numerical magnitudes.


Read more

Start Selling to Enterprises with Just a Few Lines of Code

Implement features like single sign-on in minutes instead of months.

  • WorkOS provides a complete User Management solution along with SSO, SCIM, RBAC, and audit trails.

  • Modular and easy-to-use APIs allow integrations to be completed in minutes instead of months.

  • Design and user experience are everything. From the quality of our documentation to how your users onboard, we remove all the unnecessary complexity for developers.

  • Best of all, User Management is free up to 1 million MAUs.

WorkOS powers some of the fastest growing AI startups like Perplexity, Copy ai, Jasper, and Adept.

Start integrating today

partner with us

TRENDING SIGNALS

OpenAI

OpenAI released improved version to the GPT-4o, named `chatgpt-4o-latest` and ranked 1st in lmsys arena

⇧ 2027 Likes

Language Models

XAI released early preview of Grok-2, performing a tad below Sonnet and GPT-4o, and almost tying with Llama-405B

⇧ 5300 Likes

Google

Google announced Gemini Live, a mobile conversational experience with Gemini and 10 new voices

⇧ 3417 Likes

AI Software Engineer

Cosine released Genie, an AI Software Engineer scoring a new high of 30.08% (57% improvement over prev) on SWE-Bench

⇧ 2745 Likes

AI Search

Perplexity and Polymarket team up for enhanced search results, paired with real-time probability predictions

⇧ 2371 Likes

Leading speech AI on developer-first API

AssemblyAI’s developer-first API offers the most robust set of capabilities including speech-to-text, speech understanding, and full access to Claude 3 models for precise call insights, summarizations, and more.


They lead the market with:
• The highest accuracy rates — 95%
• The lowest WER — 4.8%
• 30% less hallucinations than other providers

Get $50 credit to start building ↗️

TOP PAPERS

LLM Scaling

Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

⇧ 1120  Likes

Problem
LLMs struggle with complex reasoning tasks. Scaling model parameters is costly. The paper investigates if optimizing test-time compute can improve performance more efficiently than increasing model size.


Solution
The authors propose "compute-optimal" scaling of test-time computation. They explore two approaches: (1) searching against process-based verifier reward models, and (2) updating the model's response distribution adaptively. They develop strategies that allocate test-time compute based on prompt difficulty.


Results
On the MATH benchmark using PaLM 2-S*, compute-optimal scaling outperforms best-of-N baselines by 4x in efficiency. In FLOPs-matched comparisons, test-time compute outperforms a 14x larger model on easy/medium questions. However, for very difficult questions, scaling model parameters remains more effective.

LLM Education

Transformer Explainer: Interactive visualization tool designed for non-experts to learn about Transformers

⇧ 1050 Likes

Problem
Transformers, especially in text generation, are complex and opaque to non-experts. Existing resources often overwhelm beginners with mathematical details or focus on advanced interpretability, creating a gap in accessible educational tools for understanding Transformer models.


Solution
TRANSFORMER EXPLAINER is an open-source, web-based tool that visualizes GPT-2's text generation process. It uses Sankey diagrams to show data flow, offers multi-level abstractions, and enables real-time experimentation with model parameters. The tool runs a live GPT-2 instance in the browser, requiring no installation.

Results
The tool is available here, and also they have published a video demo for the tool's usage.

RAG

HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction

⇧ 685 Likes

Problem
Existing Retrieval Augmented Generation (RAG) techniques struggle with financial documents due to specialized terminology and complex formats. VectorRAG lacks precision in retrieving relevant context, while GraphRAG underperforms in abstractive questions without explicit entities.

Solution
HybridRAG combines VectorRAG and GraphRAG, leveraging both vector databases and knowledge graphs. It uses GPT-3.5-turbo for generation, text-embedding-ada-002 for embeddings, and a custom knowledge graph construction method. HybridRAG concatenates contexts from both approaches to generate responses.

Results
HybridRAG demonstrates superior performance in extracting information from financial documents, balancing high-quality answers with comprehensive context retrieval. Outperforms VectorRAG and GraphRAG.


Faithfulness: 0.96 (tied with GraphRAG, vs 0.94 for VectorRAG)
Answer Relevancy: 0.96 (vs 0.91 for VectorRAG, 0.89 for GraphRAG)
Context Recall: 1.0 (tied with VectorRAG, vs 0.85 for GraphRAG)

PYTHON TIP

How to Quantize Llama 3.1 with Intel's AutoRound library

Quantization reduces model size and improves inference speed by converting weights to lower precision (e.g., 4-bit).


Benefits include reduced memory footprint, faster inference, and lower compute-cost. Its very important in scenarios, such as edge computing, and server-side deployment where efficiency is crucial.


AutoRound  library implements an advanced quantization algorithm for low-bits LLM inference. It uses sign gradient descent to fine-tune rounding and minmax values of weights, achieving competitive results in just 200 iterations.


AutoRound is a very versatile library and supports different devices (CPU, GPU, Intel Gaudi2) and offers features like mixed bit-widths, group sizes, and lm-head quantization.


The code below quantizes a Meta-Llama-3.1-8B model to 4bit.


As a first step, note, Meta-Llama-3.1-8B is a gated model, so you need to request for access  to the model, from Huggingface, and normally the access is granted within 10-15 minutes.



# Install required libraries
pip install --upgrade transformers auto-round

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load the pre-trained model and tokenizer
model_name = "meta-llama/Meta-Llama-3.1-8B"
model = AutoModelForCausalLM.from_pretrained(model_name,
                        torch_dtype=torch.float16)
tokenizer = AutoTokenizer.from_pretrained(model_name)

from auto_round import AutoRound

# Set quantization parameters
bits, group_size, sym = 4, 128, True

# Initialize AutoRound with model and quantization settings
autoround = AutoRound(model, tokenizer, bits=bits,
                      group_size=group_size,
                      batch_size=2, seqlen=512, sym=sym,
                      gradient_accumulate_steps=4, device='cuda')

# Perform quantization
autoround.quantize()

# Save the quantized model to an output directory of your choice
output_dir = "./AutoRound/GPTQ-sym/"
autoround.save_quantized(output_dir)


Thank You.

Want to promote your company, product, service, or event to 190,000+ AI researchers and engineers?

Sponsor