Skip to main content
Back

A FULLY Open-Source Model Beats LLama-3.1

TRY OLMO 2

December 2, 2024 4 min read

Hey,

Welcome to AlphaSignal – the most read newsletter by AI developers. 


We bring you the top 1% of news, papers, models, and repos, all summarized to keep you updated on the latest in AI.

IN TODAY'S SIGNAL

Read time: 4 min 47 sec

🎖️ Top News

📌 Encord

⚡️ Trending Signals

📝 Top Repos

  • Qwen-Agent: Deploy LLM-powered agents with Qwen models achieving over 95% tool accuracy.

  • Docling: Transform documents to JSON/Markdown with 95% accuracy, processing 100+ pages/sec.

  • Khoj: Open-source AI assistant integrates with Obsidian, Emacs, WhatsApp, and browsers.

🧠 Tutorial

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

Open Source

OLMo 2: The Best Fully Open Language Model to Date

⇧ 1052 Likes

What's New

Allen AI released OLMo 2, a new family of fully open language models, including 7B and 13B parameter versions trained on up to 5 trillion tokens.


You can download the models, training data, and code to reproduce the results or extend them. These models outperform other open options and compete with open-weight models like Llama 3.1 8B.


What’s Available

  • Weights for base and Instruct models: OLMo 2 models on HuggingFace.

  • Datasets: Pretraining (OLMo-Mix-1124, Dolmino-Mix-1124) and post-training (Tülu 3 SFT, RLVR mix).

  • Code: Training and post-training recipes on GitHub.

What It’s Good At

  • Instruction-Following: OLMo 2 Instruct models perform well on tasks requiring clear, structured outputs, such as QA, summarization, and reasoning.

  • Reasoning and Math: Benchmarks like GSM8K and MATH show strong results, outperforming models like Llama 3.1 8B and Qwen 2.5 14B.

  • Open-Source Research and Reproducibility: Fully open datasets and training recipes make it ideal for experimenting, finetuning, or building derivative models.

What It’s Bad At

  • General-Purpose Chat: OLMo 2 lacks advanced conversational fine-tuning compared to proprietary models like ChatGPT or Claude.

  • Low-Resource or Niche Domains: The data focuses on high-quality web and academic content, which may limit performance on specialized domains without further fine-tuning.
  • Inference Efficiency: Higher computational cost for inference compared to smaller, optimized models like Mistral 7B.

OLMo 2 7B

  • ARC Challenge: 63.7 (vs. Llama 3.1 8B at 60.5).
  • MMLU: 50.2 (vs. Llama 3.1 8B at 48.6).
  • TriviaQA: 79.5 (vs. Llama 3.1 8B at 77.8).
  • GSM8K (Math): 38.1 (vs. Llama 3.1 8B at 35.3).

OLMo 2 13B

  • ARC Challenge: 67.4 (vs. Qwen 2.5 14B at 64.8).
  • MMLU: 54.1 (vs. Qwen 2.5 14B at 52.7).
  • TriviaQA: 82.3 (vs. Qwen 2.5 14B at 80.6).
  • GSM8K (Math): 46.2 (vs. Qwen 2.5 14B at 43.7).

Key Innovations in Training

OLMo 2 improves stability and efficiency in long training runs:

  • Replaces nonparametric layer norms with RMSNorm and reorders layers.

  • Uses QK-Norm and rotary positional embeddings.

  • Adds Z-loss regularization to stabilize gradients.

  • Optimizes initialization to preserve scale across layers.

Try olmo 2

Get High Quality Labeled AI Data 10x Faster

Free up your ML team's time by outsourcing large-scale data labeling projects to domain experts from any industry including healthcare, life sciences to legal and financial sectors, and more.

How it works:

  1. Customize labeling pipelines: Configure the optimal data annotation workflow to suit your project needs, including nested ontologies, model assisted automation and review stages with assistance from annotation experts at Encord. 

  2. Select on-demand specialists: Connect with our global community of specialized data labelers, selected specifically for your use case.

  3. Monitor performance in real-time: Track workforce productivity and labeling accuracy through advanced reporting dashboards. Identify and resolve bottlenecks in real-time to ensure efficient workflows and meet project deadlines with ease.

You can accelerate your AI development with Encord's outsourced labeling network and comprehensive annotation platform.

Get started

partner with us

TRENDING SIGNALS

PDF Extraction

NVIDIA releases a powerful PDF extraction workflow that can extract insights from text, graphs, charts, and tables no matter the size

⇧ 3247 Likes

OpenAI

Elon Musk files an injunction to stop OpenAI from becoming a for-profit

⇧ 529 Likes

Writing

Anthropic adds new personalization feature to Claude: it can now match your unique writing style

⇧ 2,961 Likes

Decentralization

PRIME Intellect releases INTELLECT-1: The first 10B parameter language model collaboratively trained across the globe

⇧ 569 Likes

AI Agents

You can now scrape the web agentically with Firecrawl and LangGraph (notebook)

⇧ 361 Likes

TRENDING REPOS

AI Agents

Qwen-Agent

☆ 4,220 Stars

You can build and deploy LLM-powered agents using Qwen models (7B/14B/72B). It helps you create assistants that follow instructions, use tools, plan actions, and maintain memory. The framework runs on top of Qwen's function calling capabilities with 95%+ accuracy on tool use benchmarks.

PDF Transformation

Docling

☆ 12,130 Stars

A document parsing library that transforms PDFs, docs, slides, and spreadsheets into structured Markdown and JSON. It parses complex layouts with 95% accuracy using ML, extracts tables, images, and text while preserving reading order. Processes 100+ pages/sec on CPU. Integrates with LlamaIndex/LangChain for RAG, supports OCR, and runs cross-platform via Python or CLI.

AI Assistant

Khoj

☆ 16,191 Stars

Khoj is an open source personal AI assistant that helps you chat with your documents and the web using any LLM. You can run it locally or in the cloud. Integrates with Obsidian, Emacs, WhatsApp, and browsers. 

TUTORIAL

Prompt Engineering

How to Optimize Your Prompts with Anthropic's Prompt Improver

Anthropic’s Prompt Improver refines AI prompts to enhance accuracy, reliability, and task-specific performance.


It uses techniques like chain-of-thought reasoning, example enrichment, and structural rewriting to make prompts clearer and more effective.


Testing shows it can boost task accuracy by 30% and ensure 100% adherence to word count instructions.


Step-by-Step

  1. Access the Console
    Visit Anthropic’s Console and log in.

  2. Navigate to Prompt Improver
    Select "Improve an existing prompt" from the menu.

  3. Prepare Your Prompt
    Create a reusable prompt template. Include at least one as a placeholder.

  4. Optimize Your Prompt
    Click Improve Prompt. The tool rewrites, augments examples, and applies best practices to strengthen performance.

  5. Evaluate and Iterate
    Use the Ideal Output column to grade outputs on a 5-point scale. Provide feedback to refine results.

  6. Add and Edit Examples
    In the Workbench, structure input/output pairs for increased response accuracy and consistency.

Access the console

At Alpha Signal, our mission is to build a sharp, engaged community focused on AI, machine learning, and cutting-edge language models, helping over 200,000 developers stay informed and ahead. We’re passionate about curating the best in AI, from top research and trending technical blogs to expert insights and tailored job opportunities. We keep you connected to the breakthroughs and discussions that matter, so you can stay in the loop without endless searching. We also work closely with partners who value the future of AI, including employers and advertisers who want to reach an audience as passionate about AI as we are.


Our partnerships are based on shared values of ethics, responsibility, and a commitment to building a better world through technology.Privacy is a priority at Alpha Signal. Our Privacy Policy clearly explains how we collect, store, and use your personal and non-personal information. By using our website, you accept these terms, which you can review on our website. This policy applies across all Alpha Signal pages, outlining your rights and how to contact us if you want to adjust the use of your information. We’re based in the United States. By using our site, you agree to be governed by U.S. laws.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Work with us