Hey | Welcome back to AlphaSignal, where we bring you the latest developments in the world of AI.
In the past few days 1,500+ AI papers have been released, and among them, our model has picked the six.
Lets now dive straight into the papers!
Lior |
|
|
|
IN TODAY'S SUMMARY | Reading time: 6 min 06 sec |
|
|
|
TOP PUBLICATION | Generative Multimodal Models are In-Context Learners | Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao | | Project |  | What's New | The main objectives of this work include demonstrating that large multimodal models can enhance their task-agnostic in-context learning capabilities through effective scaling-up. | Problem | The central problem tackled is the current multimodal systems' struggle to imitate the human ability to easily solve multimodal tasks in context (i.e., with only a few demonstrations or simple instructions). | Solution & Methodology | Emu2, a 37B generative multimodal model, trained on large-scale multimodal sequences with a unified autoregressive objective, is proposed. Emu2 consists of a visual encoder/decoder, and a multimodal transformer. Images are tokenized with the visual encoder to a continuous embedding space, interleaved with text tokens for autoregressive modeling.
Emu2 is first pretrained only on the captioning task with both image-text and video-text paired datasets. Emu2’s visual decoder is initialized from SDXL-base, and can be considered a visual detokenizer through a diffusion model. Here, VAE is kept frozen while the weights of a diffusion U-Net are updated. Emu-chat is derived from Emu by fine-tuning the model with conversational data, and Emu-gen is fine-tuned with complex compositional generation tasks. | Results | Results indicate that Emu2 achieves state-of-the-art few-shot performance on multiple visual question-answering datasets and demonstrates a performance improvement with an increase in the number of examples in context. Emu2 also learns to follow visual prompting in context, showcasing strong multimodal reasoning capabilities for tasks in the wild. When instruction-tuned to follow specific instructions, Emu2 further achieves new benchmarks on challenging tasks such as question answering for large multimodal models and open-ended subject-driven generation. | | Read the paper |
|
|
|
 | Build AI Solutions in Hours not Weeks | Facing an AI challenge with limited data and time?
webAI's Navigator is a new IDE crafted by AI and ML experts to streamlines the MLOps process and accelerate project completion from months to days.
The IDE offers: Streamlined Production: Full code or drag-and-drop for smooth development-to-production transition.
Advanced AI: Deep Detection, Attention Steering for object detection, conversational agents.
Full Data/Model Ownership: Retain total control over your models and data . Privacy-Secure Local Training: Train models locally for enhanced data security.
Flexible Deployment: Suitable for edge, cloud, and diverse project environments. | | Get early access → |
|
|
|
TOP PUBLICATION | Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models | Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, Karsten Kreis | | Project |  | What's New | The primary objective is to create vivid, detailed, and 3D-consistent dynamic scenes using a compositional generation framework. This method, named Align Your Gaussians (AYG), leverages dynamic 3D Gaussian Splatting with deformation fields as the 4D representation. | Problem | Diffusion models have been successful at text-to- synthesis, but have been heavily underexplored in 4D synthesis, mostly due to the extreme difficulty of the problem. The central problem addressed is the generation of dynamic, animated 3D objects from text, which is underexplored compared to static 3D object synthesis. AYG aims to synthesize dynamic 4D content using score distillation methods with an additional temporal dimension. | Solution & Methodology | The proposed solution involves a two-stage process. The first stage generates a static 3D scene using text-guided multiview diffusion models, and the second stage introduces dynamics to this static scene through a deformation field. Notably, both methods are based on variants of score distillation sampling (SDS), and the 4D dynamic scene is represented as Gaussian point clouds. The method combines text-to-image, text-to-video, and 3D-aware multiview image diffusion models to provide feedback during 4D object optimization.
The first stage is a variant of MVDream with Gaussian splatting. On top of the representation obtained in the first stage, a deformation field is learned in the second stage, where the parameters are fitted with gradients from SDS of a text-to-video diffusion model. To ensure high-quality visual content, gradients from text-to-image are also incorporated into random renderings. Optimization of deformation fields can be done autoregressively to yield long videos of 4D dynamics. | Results | The results show that AYG can generate diverse, detailed, and dynamic 4D scenes, achieving state-of-the-art text-to-4D performance. The paper also includes a comprehensive user study comparing AYG to a previous method, MAV3D, with AYG outperforming MAV3D in terms of overall quality, 3D appearance, motion amount, and motion realism. | | Read the paper |
|
|
|
TOP PUBLICATION | Gemini: A Family of Highly Capable Multimodal Models | Gemini Team Google: RohanAnil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac... | | Project |  | What’s New | Google DeepMind has used a large language model to crack a famous unsolved problem in pure mathematics.
DeepMind's FunSearch, a novel AI framework, has successfully solved the cap set problem in mathematics and enhanced bin-packing algorithms, using a unique pairing of a pre-trained Large Language Model with an automated evaluator for iterative solution development.
FunSearch is built on a modified version of Google's PaLM 2, termed Codey, optimized for code generation. It fills in missing solution components in a Python-sketched problem. | Problem | The central problem addressed by Gemini is the challenge of creating models that can effectively understand and process multiple modalities (text, image, audio, and video) while also delivering advanced reasoning and understanding in each individual domain. | Solution & Methodology | Gemini models are built on Transformer decoders. The Gemini family comprises three main sizes: Ultra: Designed for highly complex tasks, delivering state-of-the-art performance across a range of tasks. Pro: Optimized for performance and deployability at scale with strong reasoning and multimodal capabilities. Nano: Engineered for on-device applications with impressive capabilities in reasoning, STEM, coding, multimodal, and multilingual tasks.
Gemini models are built on Transformer decoders, enhanced for stable training at scale and optimized inference on Google's Tensor Processing Units. They support a 32k context length and employ efficient attention mechanisms. The models are trained to accommodate textual input interleaved with a wide variety of audio and visual inputs. The models use the SentencePiece tokenizer, which is trained on a large sample of the entire training corpus. This approach improves the inferred vocabulary, leading to better model performance, especially in tokenizing non-Latin scripts, which enhances model quality and efficiency in training and inference. | Results | Ultra: Achieved new state-of-the-art results in 30 of 32 benchmarks, including human-expert performance on MMLU. Pro: Exhibited strong performance across a wide range of tasks. Nano: Excelled in on-device tasks and showed strong performance relative to their sizes.
The models demonstrated impressive crossmodal reasoning capabilities, performing well in tasks requiring fine-grained understanding of inputs and combining information across different modalities. | | Read the paper |
|
|
|
| Outlines advancements in representing atomic systems as geometric graphs in 3D space, focusing on Geometric Graph Neural Networks (GNNs). It covers the transformation of geometric attributes according to physical symmetries like rotations and translations. The paper provides a detailed overview of Geometric GNN architectures, categorizing them as invariant networks, equivariant networks (in both Cartesian and spherical bases), and unconstrained networks. It also discusses key datasets, applications, and future research directions, aiming to make the field accessible to newcomers and provide insights into its mathematical concepts. |
| |
|
|
| Pixel-Aligned Language Model (PixelLLM) enhances large language models with the ability to perform localization tasks, using either location data as input or generating it as output. Pre-trained on Localized Narrative dataset, it aligns words with pixel locations for tasks like location-conditioned captioning and dense word grounding. PixelLLM achieves state-of-the-art results on RefCOCO and Visual Genome, demonstrating its effectiveness in location-aware vision-language tasks. |
| |
|
|
| "COLMAP-Free 3D Gaussian Splatting enables novel view synthesis and camera pose estimation without pre-computed camera poses, leveraging sequential input frame processing and explicit point cloud representations. This approach significantly improves view synthesis under large motion changes, reducing training time to around 2 hours compared to 20 hours for previous methods."
|
| |
|
|
| | | Want to promote your company, product, job, or event to 150,000+ AI researchers and engineers? You can reach out here. | |
|
|
|
|