Skip to main content
Back

Google's Latest Breakthroughs: AI Agents, Video Generation, and more

READ ALL ANNOUNCEMENTS

May 15, 2024 6 min read

Hey,

Welcome to today's edition of AlphaSignal. 


Whether you are a researcher, engineer, developer, or data scientist, our summaries are there to keep you up-to-date with the latest breakthroughs in AI. 


Let's get into it!


Lior

IN TODAY'S SIGNAL

  • Top Announcement: Google unveils Astra, Veo, 2M context Gemini, and more

  • 5 Trending Signals

  • Top repos:

    • efficient-kan

    • skyvern

    • gpt-researcher

Read Time: 6 min 32 sec

Enjoying this newsletter?
Please forward it to a friend or colleague. It helps us keep this content free.

TOP NEWS

Industry

Google IO: Google unveils AI Agent project Astra, Video Generation Model Veo, 2M token Gemini 1.5 Pro and more

What's New

Just one day after OpenAI's demonstration of GPT-4o, it's now Google's turn to present what they had quietly been working on.


Among other announcements, the Google IO conference introduced:

  • Veo: their most capable video generation model

  • Project Astra: their new project focused on building a future AI assistant

  • Updates to Gemini 1.5 Pro: two new versions of the flagship model, one that's more light-weight, the other with a 2M token context length

Let's go through each of these announcements one by one. 


Veo

Veo is Google DeepMind's most capable video generation model to date. It generates videos:

  • of high-quality with a1080p resolution

  • that can go over a minute

  •  in a wide range of cinematic and visual styles

Veo can take as input an image or a video along with a textual prompt. It can animate the image or edit the video when passed in the input.


In addition, it supports masked editing, enabling changes to specific areas of the video when you add a mask area to your video and text prompt.


When it comes to technical details, Google shared that they added more details to the captions of each video in Veo's training data. The model uses high-quality, compressed representations of video (also known as latents) to improve performance, generation speed and efficiency. 


Project Astra

Astra is Google's new project focused on building a future AI assistant, very similar to OpenAI's GPT-4o that was showcased live yesterday. 


Google's new assistant is powered by Gemini and supports audio, text, video and image shared in real-time. This project is still presented by Google as a prototype, and the capabilities of Astra were only shared through pre-recorded videos since it is still not available to all users. 


Early testers report a longer latency, and less emotional intelligence and tone for Astra compared to GPT-4o, but strong text to speech and potentially better ongoing video a long context support.


Gemini 1.5 Pro

Google unveiled two iterations of their flagship model Gemini 1.5 Pro.

  • Gemini 1.5 Pro Flash is the light-weight, fast and cost-efficient version of the model, meaning it is also multimodal and has a 1M token context length. The performance cost is small, with an MMLU of 78.9% compared to 81.9% for the original Gemini 1.5 Pro model.

  • Gemini 1.5 Pro had its context length doubled to 2M tokens. The new model is available via a waitlist for select developers building through the API.

Other announcements

Google also unveiled:

  • Imagen 3, their most capable image generation model, which will be available in multiple versions, each optimized for different types of tasks, from generating quick sketches to high-resolution images.

  • Gemma 2 and PaliGemma, two new open-source models added to the Gemma family. PaliGemma is Google's first vision-language open-source model and it is available now. Gemma 2 is a 27B parameter model that outperforms the previous version and will be available starting in June. 

The 2-hour sessions was very dense in product updates and announcements across the Google stack and products, including improvements across Search, Workspace, Photos, Android and more.

Access

The Gemini API and Google AI Studio are now available in 200+ countries, Gemini 1.5 Flash costs $0.35 per 1M tokens, with context caching coming next month.


Veo, Astra and the 2M context version of Gemini 1.5 Pro are not available for now, but you can join the waitlist to get access. 


However, Gemini 1.5 Pro Flash is available now through the API and PaliGemma is openly released on Kaggle. 

Why it Matters

Many product or project launches at the Google IO conference closely matched an OpenAI product. Namely, Veo is now OpenAI's Sora competitor, along with Astra that is GPT-4o's counterpart. 


Google still seems to be in the AI race, although we still need to gain access and test the models and products they introduced to get a better sense of their performance. 

Community Feedback

Jim Fan
Google I/O. Some thoughts: The model seems to be multimodal in, but not multimodal out. Imagen-3 and music gen models are still detached from Gemini as standalone components. Merging all modality I/O natively is the inevitable future
Jeff Dean (@🏡)
Gemini 1.5 Flash has really great qualities. A really good capable, natively multimodal, 1M token context window (with signup available to get access to a 2M token variant), and super lower latencies and fast response generation.
Liam Bolling
This is most competitive pricing for an LLM I’ve ever seen. $0.35? If you’re a developer, this is what you want to use. You crushed it @joshtwoodward! 👏
Read all announcements

Audio Events – An AI feature that detects and labels non-speech events within audio

Introducing Audio Events by Speechmatics *Applause*


Audio Events is a new, ground-breaking feature that uses AI to automatically detect and label non-speech events such as *music*, *laughter*, and *applause* within audio.


This innovation has the potential for users to:

  • Increase the accessibility of their media content across the globe

  • Reduce the effort of labelling non-speech events within audio

  • Automate all caption generation for more inclusive media

Audio Events is set to transform the landscape of media accessibility and analytics.


Be the first to test this AI capability for yourself.

Get early access

partner with us →

TRENDING SIGNALS

Industry

OpenAI Cofounder Ilya Sutskever and Superalignment Team Lead Jan Leike Resign

⇧ 8290 Likes

Model Training

PyTorch team achieves 1.46x speedup with <2% drop in accuracy for Vision Transformers using Block Sparsity

⇧ 783 Likes

Infra

NVIDIA's RAPIDS cuDF now integrated into Colab notebooks speeding up pandas code by 50x with zero code changes 

⇧ 1275 Likes

Open Source

Salesforce releases new state-of-the-art Instruction model based on Llama 3 8B beating GPT3.5 & Mixtral 8x7B

⇧ 220 Likes

Audio

ElevenLabs launches dubbing API enabling any developer to add audio or video translation to their product

⇧ 882 Likes

Make your Details Harder to Find Online

Every day, data brokers profit from your sensitive info—phone number, DOB, SSN—selling it to the highest bidder. Who’s buying it?


Best case: companies target you with ads.

Worst case: scammers and fraudsters. It's time you check out Incogni. It scrubs your personal data from the web, confronting the data brokers.


Help protect yourself from identity theft, spam calls, and health insurers raising your rates. 

Try Incogni now ↗️

TOP REPOS

Model Architecture

efficient-kan

☆ 2379 Stars

This repository contains an efficient implementation of Kolmogorov-Arnold Network (KAN), a new neural network architecture that was introduced by a recent paper.


The performance issue of the original implementation comes from expanding all intermediate variables to perform the different activation functions. This repository implements a solution to this efficiency gap by formulating all activation functions as a linear combination of a fixed set of basis functions. 


This reformulation can significantly reduce the memory cost and make the computation a straightforward matrix multiplication, and works with both forward and backward pass naturally.

Automations

skyvern

☆ 4871 Stars

Skyvern automates browser-based workflows using LLMs and computer vision. It provides a simple API endpoint to fully automate manual workflows, replacing brittle or unreliable automation solutions.


Instead of only relying on code-defined XPath interactions, Skyvern adds computer vision and LLMs to the mix to parse items in the viewport in real-time, create a plan for interaction and interact with them.


Skyvern was inspired by the Task-Driven autonomous agent design, one major bonus: we give Skyvern the ability to interact with websites using browser automation libraries.

Chatbots

gpt-researcher

☆ 10521 Stars

GPT Researcher is an autonomous agent designed for comprehensive online research on a variety of tasks.


GPT Researcher addresses issues of speed, determinism and reliability, offering a stable performance and increased speed through parallelized agent work, as opposed to synchronous operations.


The main idea is to run "planner" and "execution" agents, whereas the planner generates questions to research, and the execution agents seek the most related information based on each generated research question. Finally, the planner filters and aggregates all related information and creates a research report.

Ghita is an editor at AlphaSignal and AI Engineer at Clear Ventures. Experienced in Computer Vision and Language models, she holds 2 masters in Applied Mathematics.

Thank You.

Want to promote your company, product, job, or event to 180,000+ AI researchers and engineers?

Sponsor