Skip to main content
Back

The First Fully Transparent LLM?

PROJECT

December 17, 2023 4 min read

Hey 

Welcome back to AlphaSignal, where we bring you the latest developments in the world of AI.


In the past few days 1,500+ AI papers have been released, and among them, our model has picked the six.

Lets now dive straight into the papers!

Lior

IN TODAY'S SUMMARY

  • LLM360

  • COLMAP-Free 3D

  • FunSearch

  • Other notable papers

Reading time: 4 min 01 sec

TOP PUBLICATION

LLM360: Towards Fully Transparent Open-Source LLMs

Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao

Project

What's New

LLM360 introduces Amber and CrystalCoder, 7B parameter LLMs, offering full transparency in Large Language Model (LLM) training with comprehensive open-source release including training data and code.

Problem

Existing LLMs lack transparency in training processes, limiting the AI community's ability to assess reliability, biases, and replicability. This obscurity hinders collaborative progress and thorough understanding of LLM behaviors.

Solution

LLM360 tackles this by releasing two models, Amber and CrystalCoder, with all training components. This includes 1.3T and 1.4T token datasets, training code, intermediate checkpoints, and detailed logs. Training employs AdamW optimizer, mixed-precision techniques, and thorough data mix analysis for nuanced pre-training.

Results

Amber and CrystalCoder demonstrate robust performance on benchmarks like ARC and MMLU. Specifics include training on diverse datasets (e.g., RefinedWeb, StarCoder), achieving 582.4k tokens per second throughput, and detailed analysis of model behaviors like memorization across training stages.

Read the paper

Tired of explaining the same thing over and over again?

It’s time to delegate that work to AI. guidde is a GPT-powered extension that helps you explain the most complex tasks in seconds with AI generated documentation.

  • Turn boring documentation into stunning visual guides

  • Save valuable time by creating video documentation 11x faster

  • Share or embed your guide anywhere for your team to see

Simply click capture on our browser extension and the app will automatically generate step-by-step video guides complete with visuals, voiceover and call to actions.


The best part? The extension is 100% free. 

Try it here →

TOP PUBLICATION

COLMAP-Free 3D Gaussian Splatting

Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A. Efros, Xiaolong Wan

Project

What's New

The COLMAP-Free 3D Gaussian Splatting (CF-3DGS) framework efficiently synthesizes photo-realistic views without needing Structure-from-Motion (SfM) pre-processing, significantly reducing training time and enabling real-time rendering.

Problem

Traditional Neural Radiance Fields (NeRFs) methods depend heavily on accurately pre-computed camera poses, a time-consuming process prone to failure in textureless or repetitive regions, limiting their practicality in dynamic scene reconstruction.

Solution

CF-3DGS leverages video stream continuity and an explicit point cloud representation. It incrementally constructs 3D Gaussians for each frame, employing affine transformations for direct camera pose optimization. This process eliminates the necessity for pre-set camera poses, simplifying the optimization under large camera movements.

Results

Compared to previous methods, CF-3DGS shows marked improvement in view synthesis and camera pose estimation, especially under large motion changes. It achieves comparable performance to 3DGS models using COLMAP-assisted poses, validating its efficiency in handling dynamic scenes and large camera movements.  For instance, in the Tanks and Temples dataset, it achieves a mean Relative Pose Error (RPE) of 0.041 (translation) and 0.069 (rotation), and an Absolute Trajectory Error (ATE) of 0.004, which are significantly lower than baseline methods. 

Read the paper

Translate over 3 billion voices without the hassle of managing multiple APIs


Don’t restrict your audience to only the English-speaking world. Speechmatics has launched Real-Time Translation as part of its all-in-one Speech API. 


Transcribe and translate content to and from Spanish (and 45+ other languages) in real-time.  


You’ve already got the content, now it’s time to globalize its reach.


Try it, for free, no code ↗

TOP PUBLICATION

FunSearch: Making new discoveries in mathematical sciences using Large Language Model

Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R,..

What’s New

Google DeepMind has used a large language model to crack a famous unsolved problem in pure mathematics.


DeepMind's FunSearch, a novel AI framework, has successfully solved the cap set problem in mathematics and enhanced bin-packing algorithms, using a unique pairing of a pre-trained Large Language Model with an automated evaluator for iterative solution development.


FunSearch is built on a modified version of Google's PaLM 2, termed Codey, optimized for code generation. It fills in missing solution components in a Python-sketched problem.

Problem

The cap set problem, an open challenge in mathematics, involves finding the largest set of points in a high-dimensional grid where no three points align linearly. It's a complex issue representing a broader class of problems in extremal combinatorics.

Solution

The approach involved pairing a pre-trained LLM with an evaluator for program generation and refinement. FunSearch first generates a range of potential solutions. The evaluator then rigorously filters these solutions, retaining only the most accurate and viable ones. This process iteratively refines the solutions, enhancing their reliability and applicability.


“To be very honest with you, we have hypotheses, but we don’t know exactly why this works” says Alhussein Fawzi, a research scientist at Google DeepMind.

Results

FunSearch discovered the largest cap sets known in 20 years, outperforming existing computational methods. In bin-packing, it improved upon human-devised heuristics, demonstrating its efficacy in practical algorithm optimization.

Read more

NOTABLE PAPERS

Mathematical Language Models: A Survey

This survey paper presents an in-depth analysis of the role of Pre-trained Language Models (PLMs) and Large-scale Language Models (LLMs) in mathematical problem-solving and theorem proving. It categorizes research into two main areas: tasks (mathematical calculation and reasoning) and methodologies (PLMs and LLMs-based approaches).

EdgeSAM: Prompt-In-the-Loop Distillation for On-Device Deployment of SAM

EdgeSAM is the first SAM variant that can run at over 30 FPS on an iPhone 14 with minimal compromise in performance.

Dense X Retrieval: What Retrieval Granularity Should We Use?

Dense Retrieval X introduces "proposition" as a new retrieval unit, outperforming passage and sentence levels in dense retrieval tasks. This approach uses atomic text expressions, enhancing QA performance by providing condensed, relevant information.

Not Great   Good    Amazing

Want to promote your company, product, job, or event to 150,000+ AI researchers and engineers? You can reach out here.

Thank You