Hey | Welcome back to AlphaSignal, where we bring you the latest developments in the world of AI.
In the past few days 1,500+ AI papers have been released, and among them, our model has picked the six.
Lets now dive straight into the papers!
Lior |
|
|
|
IN TODAY'S SUMMARY | OLMo Corrective RAG MoE-LLaVA Other notable papers
Reading time: 4 min 29 sec |
|
|
|
TOP PUBLICATION | Open Language Model (OLMo) | Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison.. | | Project |  | What's New | AI2's OLMo introduces a 7 billion parameter model, outperforming Llama 2 in generative tasks, with comprehensive training data, code, and over 500 checkpoints per model, all under the Apache 2.0 License, promoting transparency and collaboration in AI. | Problem | The AI field lacks open, detailed models allowing comprehensive study and innovation. Researchers need full access to models, training data, and tools to explore AI's potential and understand its impact thoroughly. | Solution | AI2 developed OLMo, providing complete resources including training data, code, and model weights. They used a bias-free design, SwiGLU activation, Rotary positional embeddings, and a secure BPE-based tokenizer. The model underwent extensive training on 2 trillion tokens, ensuring robust performance and detailed checkpoints for analysis. | Results | OLMo achieved strong performance in text generation and reading comprehension, slightly leading in generative tasks like truthfulQA compared to Llama 2, but falling just short in some question-answering tasks. Its open-source nature and comprehensive documentation empower ongoing AI research and development. | | TRY OLMo |
|
|
|
 | Everything you need to build and launch AI apps in one place
| Now any developer can prototype, iterate and deploy AI solutions - not just a small team of ML engineers.
Whether solving problems with vision or LLM based solutions, you can train models, understand and tune outputs, and collect them to data and software components with ease in the Navigator AI IDE.
Full code or drag & drop. Designed for everyone.
Accelerate Your AI Projects: Get your AI models trained in minutes and tuned in hours.
Eliminate Training Costs: Train, test, and optimize models on local compute. On your device or across your company's network.
Deploy anywhere - Local edge devices or cloud-based solutions.
| | Get early access |
|
|
|
TOP PUBLICATION | Corrective Retrieval Augmented Generation | Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, Zhen-Hua Ling |  | What's New | CRAG boosts RAG robustness, outperforming standard RAG by up to 36.6% in accuracy. It uses a retrieval evaluator and web searches to enhance reliability and relevance of the retrieved documents, focusing on key information. | Problem | RAG systems often fail due to unreliable document retrieval, exposing generative models to inaccurate knowledge. This issue undermines the system's performance, especially in tasks reliant on the quality and relevance of the sourced information. | Solution | CRAG introduces a retrieval evaluator to assess and enhance document quality, triggering tailored retrieval actions. It employs web search and optimized knowledge utilization for automatic self-correction. A decompose-then-recompose algorithm filters irrelevant content, ensuring focus on crucial information, making it adaptable to various RAG-based approaches.
| Results | CRAG significantly improves RAG's performance across diverse datasets, showing up to 36.6% accuracy gain. Its adaptability and generalizability are proven through extensive testing, improving short- and long-form generation tasks. Results demonstrate CRAG's efficiency in automatic self-correction and optimizing the use of retrieved documents.
| | Read the paper |
|
|
|
TOP PUBLICATION | MoE-LLaVA: Mixture of Experts for Large Vision-Language Models | Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, Li Yuan | | Code |  | What’s New | MoE-LLaVA introduces a sparse model architecture for Large Vision-Language Models (LVLMs), achieving LLaVA-1.5-7B performance with only 3 billion sparsely activated parameters, significantly reducing computational costs without sacrificing accuracy. | Problem | Expanding LVLMs improves performance but drastically raises training and inference costs, as all parameters are used for each calculation. Multi-modal learning and model sparsity often lead to performance drops, demanding an efficient solution. | Solution | The researchers developed MoE-tuning, adapting the Mixture of Experts (MoE) architecture to LVLMs. This approach constructs a sparse model that activates only the top-k experts, maintaining computational efficiency. The training involves three stages: adapting image tokens to LLM, tuning with multi-modal instruction data, and initializing and activating the top-k experts in MoE layers. | Results | MoE-LLaVA matches the performance of LLaVA-1.5-7B across various visual understanding datasets and exceeds LLaVA-1.5-13B in object hallucination benchmarks. Training stability issues, particularly with 16-bit float precision, are noted. The framework's flexibility allows expansion to additional tasks like detection, segmentation, and accommodating more modalities. | | Read the paper |
|
|
|
| WRAP (Web Rephrase Augmented Pretraining) accelerates LLM pretraining 3x, reducing data needs by 10x through synthetic rephrases of web content. By paraphrasing the C4 dataset into diverse styles like Wikipedia, it improves perplexity and zero-shot QA accuracy. Utilizes smaller models and natural internet diversity to maintain quality and reduce bias. |
| |
|
|
| ReplaceAnything3D (RAM3D) by Meta is a text-guided 3D scene editing framework allowing object replacement in multi-view scenes for realistic integration, overcoming structural loss limitations. It showcases effectiveness in various realistic scenes, despite artifacts like the Janus multi-face problem. Future work includes geometry-appearance disentanglement and prompt-debiasing to refine scene editing. |
| |
|
|
| SliceGPT by Microsoft compresses large language models, like LLAMA2-70B, by removing up to 25% of parameters while retaining up to 99% performance. It uses dense matrix replacement for sparsification, reducing compute on GPUs to 64%-66%. Code is open source. |
| |
|
|
| | | Want to promote your company, product, job, or event to 150,000+ AI researchers and engineers? You can reach out here. | |
|
|
|
|