Skip to main content
Back

Ollama Integrates Llama Vision Models

SEE X-PORTRAIT 2 IN ACTION

November 8, 2024 5 min read

Hey,

Welcome to AlphaSignal – the most read newsletter by AI developers. 


We bring you the top 1% of news, papers, models, and repos, all summarized to keep you updated on the latest in AI.

IN TODAY'S SIGNAL

Read time: 5 min 20 sec

🎖️ Top News

📌 Web AI

⚡️ Trending Signals

📌 Vellum

  • Webinar: Super Fast Inference + Agents  

📕 Top Papers

  • MoICL reduces memory use and improves performance on noisy datasets, outperforms baselines.

  • Tool usage adaptation boosts accuracy and reduces over-reliance for scientific problem-solving models.

  • Ferret-UI 2 improves universal UI understanding across multiple platforms with better generalization.

🧠 Tutorial

  • Use Python’s caching to speed up computations by storing results of repeated function calls.

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

Expression Encoder Model

ByteDance’s X-Portrait 2 transforms static images into lifelike, expressive animations with precision

⇧ 3,598 Likes

What's New

X-Portrait 2 is an AI model from ByteDance that converts static portrait images into dynamic, expressive animations by mapping facial movements from a reference video.

This system requires only a single video to drive the animation and can transfer subtle facial expressions and complex movements like pouting, frowning, and tongue gestures. It supports both realistic portraits and cartoon images, making it applicable for animation, virtual agents, and visual effects creation.

Key Features

  • Expressive Animation Generation: Converts a single reference video and static image into an animated character, transferring detailed facial expressions.

  • High Fidelity Movement Transfer: Handles subtle expressions and complex motions (e.g., pouting, frowning, and head movements) accurately.

  • Cross-style Transfer: Can apply expression transfers to both realistic portraits and cartoon images, expanding the scope of creative use cases.

Model Performance
X-Portrait 2 outperforms previous models like X-Portrait and Runway’s Act-One, particularly in handling fast movements and small facial changes with high fidelity.

Usage

  • The system can be used to create expressive character animations for various applications such as movies, games, and virtual agents.

  • There is no indication of it being open source, nor is the model code or weights explicitly mentioned as publicly available.

  • X-Portrait 2 could be integrated into platforms like TikTok, potentially offering a free competitor to commercial avatar creation tools, but pricing details are unclear.

See x-portrait 2 in action

Local AI built for Global Enterprises

Web AI is shifting AI development by offering local solutions that move past the limits of cloud-based systems. The platform gives businesses the ability to build, deploy, and scale AI models directly on their devices.

Why Web AI?

  • Privacy: AI processing happens locally, keeping your data on your network.

  • Control: You own your AI models and data, with no reliance on third parties.

  • Real-Time, Offline: Web AI works in real-time, even without internet access, making it suitable for industries with critical operations.

  • Cost and Energy Efficiency: Skip the cloud infrastructure costs. Use your own hardware to deploy AI, cutting both expenses and energy use.

The update is available now.

Learn more

partner with us

TRENDING SIGNALS

Chatbot

Unrestricted AI group Nous Research unveils open-source Hermes 3-70B chatbot with new features

⇧ 938 Likes

Agent Memory

Andrew Ng announces course: focuses on agent memory and persistent storage with Letta AI

⇧ 1,542 Likes

AI Safety

Mistral AI releases content moderation API with multilingual LLM classifier for safer deployments

⇧ 724 Likes

Developer Tool

Google AI Studio enables automatic import of Gemini API keys into Colab for faster integration

⇧ 1,029 Likes

LLM

LlamaIndex introduces chat-ui React library for customizable LLM chat interfaces with advanced features

⇧ 696 Likes

Webinar: Super Fast Inference + Agents

If you're building agents and want to speed up inference time, you should join this GenAI webinar. Learn how to run Llama 3.1 70b at 2,100 t/s and explore agentic use cases for due diligence and research. 

Grab your spot today ↗️

TOP PAPERS

Model Optimization

Mixtures of In-Context Learners

⇧ 2821 Likes

Problem

In-context learning (ICL) allows LLMs to adapt without fine-tuning but suffers from increased memory complexity and inefficiency. It treats all demonstrations equally, leading to quadratic growth in model size and slower inference times, especially with noisy or imbalanced data.

Solution
MoICL addresses these issues by treating subsets of demonstrations as experts. It learns a weighting function to merge expert outputs, improving performance while reducing the number of demonstrations needed. This method also mitigates the negative impact of noisy, out-of-domain, or imbalanced data.

Results
MoICL outperforms strong baselines, showing up to 13% improvement on classification tasks. It also reduces inference time and memory usage, achieving comparable or better results with fewer demonstrations. MoICL demonstrates robustness to data variability, offering up to 49% improvement in handling imbalanced or noisy datasets.


LLM

Adapting While Learning: Grounding LLMs for Scientific Problems with Intelligent Tool Usage Adaptation

⇧ 908 Likes

Problem

LLMs struggle with complex scientific problems, often generating hallucinations. Integrating tools can improve accuracy, but it leads to over-reliance on tools, reducing the model's ability to solve simpler problems independently. Current methods fail to balance tool usage with basic reasoning.

Solution
We introduce a two-component fine-tuning method: World Knowledge Distillation (WKD) and Tool Usage Adaptation (TUA). WKD helps the model internalize domain knowledge from tool-generated solutions. TUA partitions problems into easy and hard categories, training the model to switch to tools only for complex tasks.

Results
Our method improves answer accuracy by 28.18% and tool usage precision by 13.89% across six scientific benchmarks, including mathematics, climate science, and epidemiology. It outperforms state-of-the-art models like GPT-4 and Claude-3.5.

Multimodal ML

Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms

⇧ 1873 Likes

Problem

Creating a universal UI understanding model is difficult due to challenges like platform diversity, resolution variation, and data limitations. Existing models often struggle to generalize across different devices, making them less adaptable to a wide range of user interfaces.

Solution
Ferret-UI 2 is a multimodal large language model (MLLM) designed for universal UI understanding. It supports multiple platform types (iPhone, Android, iPad, Webpage, AppleTV), offers high-resolution perception with adaptive scaling, and uses GPT-4o-powered task training data generation with set-of-mark visual prompting.

Results
Ferret-UI 2 outperforms its predecessor, Ferret-UI, in tasks like referring, grounding, and user-centric interactions across 9 subtasks and 5 platforms. It also demonstrates strong cross-platform transfer capabilities and performs well on benchmarks like the GUIDE next-action prediction dataset and GUI-World multi-platform benchmark.

PYTHON TIP

Leverage Caching for Faster Computations in Python

The '@lru_cache' decorator from Python’s functools can significantly speed up functions with repetitive computations by caching results. When you call a function with the same arguments, '@lru_cache' fetches the result from memory instead of recalculating it.


When to Use

If your function relies on expensive operations (e.g., API calls or recursive calculations) and is called repeatedly with identical inputs.


Benefits

Reduces redundant computations, optimizing runtime in models or data processing pipelines.


from functools import lru_cache

@lru_cache(maxsize=128)
def expensive_function(x):
    # Simulate an expensive computation
    return x ** 2

print(expensive_function(10))  # Cached for future calls with x=10



At Alpha Signal, our mission is to build a sharp, engaged community focused on AI, machine learning, and cutting-edge language models, helping over 200,000 developers stay informed and ahead. We’re passionate about curating the best in AI, from top research and trending technical blogs to expert insights and tailored job opportunities. We keep you connected to the breakthroughs and discussions that matter, so you can stay in the loop without endless searching. We also work closely with partners who value the future of AI, including employers and advertisers who want to reach an audience as passionate about AI as we are.


Our partnerships are based on shared values of ethics, responsibility, and a commitment to building a better world through technology.Privacy is a priority at Alpha Signal. Our Privacy Policy clearly explains how we collect, store, and use your personal and non-personal information. By using our website, you accept these terms, which you can review on our website. This policy applies across all Alpha Signal pages, outlining your rights and how to contact us if you want to adjust the use of your information. We’re based in the United States. By using our site, you agree to be governed by U.S. laws.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Work with us