Skip to main content
Back

Microsoft's Reveals 1Bit LLMs

READ PAPER

March 1, 2024 3 min read

Hey,

Welcome to this week's edition of AlphaSignal. 


Whether you are a researcher, engineer, developer, or data scientist, our summaries ensure you're always up-to-date with the latest breakthroughs in AI. 


Let's get into it!


Lior

IN TODAY'S EMAIL

  • Microsoft's Introduces 1-bit LLMs

  • Top News/Announcements

  • Implementing GPT in 60 Lines of NumPy

  • Alibaba's EMO: Emote Portrait Alive

Read Time: 3 min 27 sec

TOP PAPER

Microsoft's Introduces 1-bit LLMs

What's New?

Microsoft released BitNet b1.58, a new Large Language Model (LLM) using 1.58 bits per parameter, reducing computational demands significantly while maintaining performance. 


Unlike traditional 16-bit models, BitNet employs ternary values (-1, 0, 1), slashing GPU memory use and energy consumption by up to 3.5 times and 71 times respectively, without sacrificing model accuracy.


BitNet b1.58 achieves comparable or superior results to FP16 models like LLaMA 3B in perplexity and various language tasks starting from a 3 billion parameter size. 


The reduction in precision to 1-bit minimizes the need for energy-heavy floating-point operations, particularly in matrix multiplication, speeding up computations and lowering energy costs.


This model's efficiency improves as it scales, offering significant performance enhancements, especially in larger models. For instance, at 70 billion parameters, BitNet is over four times faster than traditional models, increasing throughput and reducing latency.

Read paper

Learn from top GenAI experts at GenAI Productionize 2024

See how LlamaIndex, Coinbase, LinkedIn, Comcast, Proctor & Gamble, Roblox, Databricks, JPMorgan Chase, Fidelity, Chegg and more get their GenAI apps into production, including practical strategies for governance, evaluation, and monitoring.


Register for GenAI Productionize 2024 to learn:
- How organizations have successfully adopted enterprise GenAI
- Practical hands-on architecture and tool insights from leading GenAI builders
- The emerging enterprise GenAI stack, from orchestration to evaluation, inference, retrieval and more

Register now

partner with us

TOP NEWS

Figure AI partners with OpenAI and raises $675 million to develop models for humanoid robots

---------------------

OpenAI and Figure AI collaborate to create advanced AI models for humanoid robots, aiming to speed up market readiness by teaching them natural language understanding for logical reasoning. Figure AI secured $675 million funding at a $2.6 billion valuation, backed by tech giants including Microsoft and Nvidia, and individuals like Jeff Bezos.

DeepLearning AI and Meta release a course on prompt engineering with Llama 2

---------------------

The free short course on Prompt Engineering with Llama 2, focuses on best practices for interacting with its collection of models including Chat Llama, Code Llama, and Llama Guard for safe AI applications, emphasizing prompt crafting and application development.

LLaMa 3 announced for July 2024

---------------------

Meta's Llama 3, set for July release, aims to rival GPT-4 with over 140 billion parameters, potentially multimodal capabilities. It seeks improved responsiveness and nuanced safety measures, addressing complex queries with context. Llama 2, its predecessor, had up to 70 billion parameters.

TOP PAPER

EMO: Emote Portrait Alive

Repo

What's New?

Alibaba recently announced EMO, a model capable of generating realistic talking and singing videos from any image or audio. 


This technology leverages a two-step process. Initially, it analyzes the photo to identify facial movements through Frames Encoding. Then, it processes the audio to map appropriate facial expressions to the voice and mood using a Diffusion Process, enhancing the realism of expressions in sync with audio inputs.


EMO stands out for its ability to animate portraits and drawings, adapting to fast songs and maintaining the character's identity across various expressions and languages. It utilizes a novel diffusion model approach, focusing on the nuances of human facial dynamics without the need for 3D graphics or traditional animation techniques. 


Key components include an audio encoder for motion generation, a reference encoder for visual identity, temporal modules for smooth transitions, and facial masks to detail core face regions.


Trained on a vast dataset, EMO has shown superiority over previous methods like Wav2Lip and DreamTalk in metrics such as Fréchet Inception Distance, expression modeling, and lip sync accuracy. 

Read paper

TUTORIAL

Implementing GPT in 60 Lines of NumPy

What's New?

This tutorial provides a step-by-step guide to creating a simplified GPT model using only 60 lines of NumPy. It assumes familiarity with Python, NumPy, and basic neural network training concepts. 


The focus is on constructing a basic yet complete version of the GPT architecture for educational purposes, highlighting key components such as input/output functions, text generation techniques, and the training process. 


It covers the GPT's fundamental elements, including embeddings, decoder stacks, and attention mechanisms. 

Read tutorial

Not Great   Good    Amazing

Thank You.

Want to promote your company, product, job, or event to 170,000+ AI researchers and engineers? You can reach out here.