Skip to main content
Back

The Largest Open LLM is Out: Grok

VIEW REPO

March 18, 2024 4 min read

Hey,

Welcome to this week's edition of AlphaSignal. 


Whether you are a researcher, engineer, developer, or data scientist, our summaries ensure you're always up-to-date with the latest breakthroughs in AI. 


Let's get into it!


Lior

IN TODAY'S EMAIL

  • xAI turns Grok open-source

  • Top Repos of the week

  • Apple releases new multimodal models

  • Top of Reddit

Read Time: 4 min 32 sec

TOP NEWS

Grok-1 is Now Open-Source: The Largest Open LLM

What's New?

xAI finally open-source Grok-1 making it the largest open LLM ever built.


With 314-billion parameters, the Mixture of Experts (MoE) model utilizes 86 billion active parameters at any given time, enhancing its processing capabilities. 


Unlike traditional models, Grok-1 employs Rotary Embeddings, avoiding fixed positional limitations and supporting a more dynamic data interpretation.


Key Specifications:

  • Parameters: 314 billion, with 25% of weights active per token.
  • Architecture: Mixture of 8 Experts, using 2 per token.
  • Layers: 64 transformer layers, integrating multihead attention and dense blocks.
  • Tokenization: Utilizes a SentencePiece tokenizer, vocab size of 131,072.
  • Embedding and Positional Encoding: 6,144 embedding size, matching rotary positional embeddings.
  • Attention: 48 heads for queries, 8 for keys/values, each with a size of 128.
  • Context Length: Capable of processing 8,192 tokens with bf16 precision.

Performance Metrics:

  • Outperforms LLaMa 2 70B and Mixtral 8x7B with a MMLU score of 73%, showcasing its efficiency and accuracy in various tests.

Implementation Details:

  • Requires significant GPU resources due to its size.
  • Uses an inefficient MoE layer implementation to avoid custom kernel needs, focusing on model correctness validation.
  • The model supports activation sharding and 8-bit quantization to optimize performance.

Open-Source Availability:

  • Released under the Apache 2.0 license, Grok-1’s weights and architecture are accessible for community use and contribution.
View repo

The container-based GPU cloud developed by Latitude.sh

Launchpad efficiently leverages every piece of the hardware to start almost instantly, bringing unrelenting performance that makes other tools feel just too slow. Effortlessly fine-tune and deploy your Machine Learning models with hourly billing starting at  $1.32 / hr.


The containers are provisioned directly on bare metal, which means their resources are fully dedicated. All containers are powered by 16 CPUs, 185 GB of RAM, 1 TB of NVMe Storage, and a single GPU, which you can choose from :

  • NVIDIA's H100 (80 GB vRAM)

  • NVIDIA's L40S (48 GB vRAM)

Join the beta

partner with us

TOP REPOS

Open-Sora (☆ 6.2k)

---------------------

Open-Sora helps you generate high-quality videos efficiently via diffusion models, providing open-source tools for data preprocessing, training with acceleration techniques like sequence parallelism, and inference. It leverages transformer architectures like STDiT, optimized components like flash attention, VAEs, and text encoding with T5. Model weights for resolutions up to 512x512 are available.

unsloth (☆ 4.6k)

---------------------

Unsloth helps you finetune large language models like Mistral, Gemma, and Llama 2-5x faster using up to 70% less memory on NVIDIA GPUs. It employs techniques like LoRA, bitsandbytes, and custom Triton kernels for efficient training acceleration without losing accuracy. It supports conversational models, raw text, and direct preference optimization (DPO) for reward modeling.

PrivateGPT (☆ 49.2k)

---------------------

PrivateGPT helps you build private, context-aware AI applications using large language models on local documents without internet. It provides an API for ingesting documents, generating embeddings, retrieving relevant context, and generating responses using retrieval-augmented generation pipelines based on LlamaIndex abstractions. It supports completions, streaming, and extends OpenAI API standards.

Add an AI assistant to your product this quarter

Enable your users to interact with your product through a conversational interface built by industry experts. Spine AI’s copilots bring workflow automation, analytics and advanced search to your product with unparalleled ease with little effort from your team.


Book a demo ↗️

TOP PAPER

Apple Releases New Multimodal Models: MM1 Family

What's New?

Apple has released MM1, a new family of multimodal AI models designed for processing visual and textual data. The release includes detailed information unusual for Apple.

  • Model Composition: MM1 encompasses models with up to 30 billion parameters, trained on data including image captions, combined image-text, and text-only datasets.

  • Learning Efficiency: The 30 billion parameter version shows strong capabilities in few-shot learning, indicating effective learning from limited examples.

  • Benchmarking: MM1 competes with existing models like GPT-4V and Gemini Pro in pre-training and fine-tuning performance.

  • Technical Details:

    • Performance increases significantly with enhancements in the image encoder and adjustments in image resolution.
    • Optimal data combination for training includes image-caption, interleaved image-text, and text-only data.
    • The vision-language connector has a smaller effect on performance compared to other factors.

  • Performance Metrics: The MM1-30B model achieves a 39.4 score in zero-shot settings and 44.4 in eight-shot settings on the MathVista benchmark, demonstrating strong few-shot and reasoning abilities.

View research

TOP OF REDDIT

Did OpenAI just accidentally leak the next big ChatGPT upgrade?

⇧ 933

You can see cached snippets in Bing and DuckduckGo.

Search for "OpenAI blog gpt-4.5 turbo"

CEO says he tried to hire an AI researcher from Meta and was told to 'come back to me when you have 10,000 H100 GPUs'

⇧ 872

Gemma finetuning should be much better now

⇧ 283

I managed to find 8 bugs in Google's Gemma implementation in multiple repos! This caused finetuning runs to not work correctly. The full list of issues include:

Must add <bos> or else losses will be very high.

There’s a typo for model in the technical report!


sqrt(3072)=55.4256 but bfloat16 is 55.5.

Layernorm (w+1) must be in float32.

Keras mixed_bfloat16 RoPE is wrong.

RoPE is sensitive to y*(1/x) vs y/x.

RoPE should be float32 - already pushed to transformers 4.38.2.

GELU should be approx tanh not exact.

Not Great   Good    Amazing

Ghita is an editor at AlphaSignal and AI Engineer at Clear Ventures. Experienced in Computer Vision and Language models, she holds 2 masters in Applied Mathematics.

Thank You.

Want to promote your company, product, job, or event to 175,000+ AI researchers and engineers? You can reach out here.