Skip to main content
Back

The Biggest Development in Open-Source LLMs?

PROJECT

February 4, 2024 4 min read

Hey 

Welcome back to AlphaSignal, where we bring you the latest developments in the world of AI.


In the past few days 1,500+ AI papers have been released, and among them, our model has picked the six.

Lets now dive straight into the papers!

Lior

IN TODAY'S SUMMARY

  • OLMo

  • Corrective RAG

  • MoE-LLaVA

  • Other notable papers

Reading time: 4 min 29 sec

TOP PUBLICATION

Open Language Model (OLMo)

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison..

Project

What's New

AI2's OLMo introduces a 7 billion parameter model, outperforming Llama 2 in generative tasks, with comprehensive training data, code, and over 500 checkpoints per model, all under the Apache 2.0 License, promoting transparency and collaboration in AI.

Problem

The AI field lacks open, detailed models allowing comprehensive study and innovation. Researchers need full access to models, training data, and tools to explore AI's potential and understand its impact thoroughly.

Solution

AI2 developed OLMo, providing complete resources including training data, code, and model weights. They used a bias-free design, SwiGLU activation, Rotary positional embeddings, and a secure BPE-based tokenizer. The model underwent extensive training on 2 trillion tokens, ensuring robust performance and detailed checkpoints for analysis.

Results

OLMo achieved strong performance in text generation and reading comprehension, slightly leading in generative tasks like truthfulQA compared to Llama 2, but falling just short in some question-answering tasks. Its open-source nature and comprehensive documentation empower ongoing AI research and development.

TRY OLMo

Everything you need to build and launch AI apps in one place

Now any developer can prototype, iterate and deploy AI solutions - not just a small team of ML engineers.

Whether solving problems with vision or LLM based solutions, you can train models, understand and tune outputs, and collect them to data and software components with ease in the Navigator AI IDE.

Full code or drag & drop. Designed for everyone.

  • Accelerate Your AI Projects: Get your AI models trained in minutes and tuned in hours.

  • Eliminate Training Costs: Train, test, and optimize models on local compute. On your device or across your company's network.

  • Deploy anywhere - Local edge devices or cloud-based solutions.

Get early access

TOP PUBLICATION

Corrective Retrieval Augmented Generation

Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, Zhen-Hua Ling

What's New

CRAG boosts RAG robustness, outperforming standard RAG by up to 36.6% in accuracy. It uses a retrieval evaluator and web searches to enhance reliability and relevance of the retrieved documents, focusing on key information.

Problem

RAG systems often fail due to unreliable document retrieval, exposing generative models to inaccurate knowledge. This issue undermines the system's performance, especially in tasks reliant on the quality and relevance of the sourced information.

Solution

CRAG introduces a retrieval evaluator to assess and enhance document quality, triggering tailored retrieval actions. It employs web search and optimized knowledge utilization for automatic self-correction. A decompose-then-recompose algorithm filters irrelevant content, ensuring focus on crucial information, making it adaptable to various RAG-based approaches.

Results

CRAG significantly improves RAG's performance across diverse datasets, showing up to 36.6% accuracy gain. Its adaptability and generalizability are proven through extensive testing, improving short- and long-form generation tasks. Results demonstrate CRAG's efficiency in automatic self-correction and optimizing the use of retrieved documents.

Read the paper

TOP PUBLICATION

MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, Li Yuan

Code

What’s New

MoE-LLaVA introduces a sparse model architecture for Large Vision-Language Models (LVLMs), achieving LLaVA-1.5-7B performance with only 3 billion sparsely activated parameters, significantly reducing computational costs without sacrificing accuracy.

Problem

Expanding LVLMs improves performance but drastically raises training and inference costs, as all parameters are used for each calculation. Multi-modal learning and model sparsity often lead to performance drops, demanding an efficient solution.

Solution

The researchers developed MoE-tuning, adapting the Mixture of Experts (MoE) architecture to LVLMs. This approach constructs a sparse model that activates only the top-k experts, maintaining computational efficiency. The training involves three stages: adapting image tokens to LLM, tuning with multi-modal instruction data, and initializing and activating the top-k experts in MoE layers.

Results

MoE-LLaVA matches the performance of LLaVA-1.5-7B across various visual understanding datasets and exceeds LLaVA-1.5-13B in object hallucination benchmarks. Training stability issues, particularly with 16-bit float precision, are noted. The framework's flexibility allows expansion to additional tasks like detection, segmentation, and accommodating more modalities.

Read the paper

NOTABLE PAPERS

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

WRAP (Web Rephrase Augmented Pretraining) accelerates LLM pretraining 3x, reducing data needs by 10x through synthetic rephrases of web content. By paraphrasing the C4 dataset into diverse styles like Wikipedia, it improves perplexity and zero-shot QA accuracy. Utilizes smaller models and natural internet diversity to maintain quality and reduce bias.

ReplaceAnything3D:Text-Guided 3D Scene Editing with Compositional Neural Radiance Fields

ReplaceAnything3D (RAM3D) by Meta is a text-guided 3D scene editing framework allowing object replacement in multi-view scenes for realistic integration, overcoming structural loss limitations. It showcases effectiveness in various realistic scenes, despite artifacts like the Janus multi-face problem. Future work includes geometry-appearance disentanglement and prompt-debiasing to refine scene editing.

SliceGPT: Compress Large Language Models by Deleting Rows and Columns

SliceGPT by Microsoft compresses large language models, like LLAMA2-70B, by removing up to 25% of parameters while retaining up to 99% performance. It uses dense matrix replacement for sparsification, reducing compute on GPUs to 64%-66%. Code is open source.

Not Great   Good    Amazing

Want to promote your company, product, job, or event to 150,000+ AI researchers and engineers? You can reach out here.

Thank You