Hey, | Welcome to today's edition of AlphaSignal.
Whether you are a researcher, engineer, developer, or data scientist, our summaries are there to keep you up-to-date with the latest breakthroughs in AI.
Let's get into it,
Lior | . |
|
|
|
IN TODAY'S SIGNAL
| Top News
Snorkel AI
Trending Signals
Top Repos
Mora: an open-source version of video generation model Sora.
Omost: converting LLMs coding abilities to image generation.
gpt-computer-assistant: GPT-4o for MacOS, Windows, Ubuntu.
Python Tip |
Read Time: 5 min 01 sec |
|
|
|
Enjoying this newsletter? Please forward it to a friend or colleague. It helps us keep this content free.
|
|
|
|
TOP ANNOUNCEMENT
| Model Architecture | Mamba-2: A new state space model architecture that outperforms Mamba and Transformer++ |
⇧ 1981 Likes |
 |
What's New | After the success of Mamba-1, which accumulated over 10k stars on GitHub, researchers Tri Dao and Albert Gu introduce Mamba-2.
Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of transformers.
You can use Mamba-2 with PyTorch to build and train neural networks handling information-dense data efficiently.
Core Innovation: Structured State Space Duality (SSD)
Mamba-2's core innovation, SSD, combines SSMs and attention mechanisms. SSD constrains the recurrent matrix 𝐴A to a scalar-times-identity structure, simplifying computations and enhancing hardware efficiency.
This design enables multi-head SSMs, increasing the state size from 16 in Mamba-1 to 64-256 in Mamba-2, and uses matrix multiplications optimized for GPUs/TPUs.
Performance Improvements
Mamba-2 trains 50% faster than Mamba-1 and handles larger state dimensions. At the 3B scale, Mamba-2, trained on 300B tokens, surpasses Mamba-1 and older transformers in performance. The model significantly outperforms Mamba-1 on tasks like multi-query associative recall (MQAR) due to its larger state sizes.
Architectural Changes Mamba-2 introduces parallel parameter generation for 𝐴A, 𝐵B, and 𝐶C, enabling tensor parallelism and scaling. It maintains efficient memory usage with a constant state size per channel and leverages matrix multiplications for faster computation. These changes simplify the architecture and improve scalability.
Empirical Results
In language modeling tasks, Mamba-2 shows slightly better scaling compared to Mamba-1, adhering to Chinchilla laws, with faster training times. Pretrained models ranging from 130M to 2.8B parameters are available, trained on datasets like Pile and SlimPajama. Performance is consistent across architectures, with minor variations in zero-shot evaluation results due to evaluation noise.
Key Specifications - State Size: Increased from 16 (Mamba-1) to 64-256 (Mamba-2)
- Training Speed: 50% faster than Mamba-1
- Scale: Models range from 130M to 2.8B parameters
- Datasets: Trained on Pile and SlimPajama
- Evaluation Tasks: Includes MQAR and zero-shot evaluations on various benchmarks
|
| Access |
Install Mamba-2 with !pip install mamba-ssm and integrate it with PyTorch.
Pretrained models can be accessed on Hugging Face for various sequence tasks.
Read the blog post |
|
Mamba repo
|
|
|
|
| New Webinar: How to align LLMs to enterprise objectives and policies |
The biggest challenge enterprise AI teams face is aligning LLMs to domain- and enterprise-specific objectives and policies – and ensuring they’re compliant with organizational standards and ethics. In this webinar,
we'll introduce an enterprise alignment workflow
show how we applied it at top finance, insurance and healthcare companies
and improved the accuracy of their AI assistants and chatbots by 20+ points Can’t attend live? All registrants will receive an on-demand recording. |
|
Register now (free)
| partner with us |
|
|
|
TRENDING SIGNALS
| Datasets |
|
⇧ 4766 Likes | |
Open Source |
| ⇧ 256 Likes |
| ChatGPT |
|
⇧ 905 Likes | |
Video Generation |
|
⇧ 1084 Likes | |
Industry |
|
⇧ 2421 Likes | | |
|
|
|
|
TOP REPOS
| Video Generation | |
☆ 1304 Stars | Mora helps you perform generalist video generation tasks using a multi-agent framework. It supports text-to-video, text-conditional image-to-video, video extension, video-to-video editing, and digital world simulation.
Mora is open-source, extendable, and achieves performance close to OpenAI's Sora in various tasks, with videos up to 80 seconds long. |
| Image Generation |
| ☆ 4478 Stars |
Omost converts LLM coding capabilities into image generation. It employs three pre-trained models based on Llama 3 and Phi 3.
These models are trained using ground-truth annotations, automatic image annotations, Direct Preference Optimization (DPO), and tuning data from OpenAI GPT-4. To deploy, you need an 8GB Nvidia GPU. | |
Assistants | |
☆ 1541 Stars | This repo helps you run the ChatGPT MacOS app on Windows and Linux. You can easily install as Python library for this time but use the provided pipeline with native install scripts.
The capabilities of this computer assistant include: - Screen Read
- Microphone
- System Audio
- Memory
| |
|
|
|
|
PYTHON TIP
| Leveraging walrus operator for concise variable assignment and conditional expression | The walrus operator (:=) allows you to assign a value to a variable within a conditional expression. By using this operator, you are ensuring:
- Improved readability: The code becomes more concise and easier to follow, especially when dealing with complex conditions.
- Reduced redundancy: You avoid repeating the variable name for assignment and checking.
Here's an example to shows you how you could use this operator: |
if number := int(input("Enter a number: "), base=10) > 0: print(f"The number is positive.") else: print(f"The number is non-positive.")
| In the code above:
- number := int(input(...)) combines the input conversion and variable assignment into a single line.
- The if statement uses the assigned number within the condition.
This approach can be particularly helpful when you need to use the assigned value immediately within the conditional check. |
|
|
|
Ghita is an editor at AlphaSignal and AI Engineer at Clear Ventures. Experienced in Computer Vision and Language models, she holds 2 masters in Applied Mathematics.
|
|
|
|
Want to promote your company, product, job, or event to 180,000+ AI researchers and engineers?
|
|
Sponsor
|
|
|
|
|