| Hey, | Welcome to today's edition of AlphaSignal, a newsletter for developers by developers.
We identify and summarize the top 1% news, papers, models, and repos in the AI industry. |
|
|
|
IN TODAY'S SIGNAL
| Read time: 4 min 54 sec | 🎖️ Top News
📌 Ori
Ori Serverless Kubernetes: AI cloud service with affordable, scalable, simple management. Pay-per-use pricing. Autoscaling, powerful GPUs. $1000 signup credit.
⚡️ Trending Signals
🛠️ Top Repos
MetaGPT: Build entire software projects with one-line.
Deep-Live-Cam: Real time face swap and one-click video deepfake.
MiniCPM-V: GPT-4V Level MLLM for Single Image, Multi-Image and Video on Your Phone
🧠 Tutorial |
|
|
|
If you're enjoying AlphaSignal please forward this email to a colleague.
It helps us keep this content free. |
|
|
|
TOP NEWS
| LLM Inference Cost | Anthropic launched prompt-caching feature with 90% cost reduction, 85% lower latency |
⇧ 14,329 Likes |
 |
What's New | Anthropic has introduced prompt caching for Claude AI models, allowing developers to reuse large portions of prompts between API calls. This slashes costs by 90% and reduces response time by 85%.
Key Features
Available for Claude 3.5 Sonnet and Claude 3 Haiku, with Claude 3 Opus support coming soon.
Ideal for Large Documents and repetitive instructions.
Supports Multi-Turn Conversations.
Pricing: Claude 3.5 Sonnet: $3/Mn tokens input, $3.75/Mn tokens cache write, $0.30/MTok cache read Claude 3 Haiku: $0.25/Mn tokens input, $0.30/Mn tokens cache write, $0.03/MTok cache read
Cache write tokens are 25% more expensive than base input tokens, while cache read tokens are 90% cheaper than base input tokens.
How It Works Developers only need to add a cache control attribute ("cache_control": ) to content they want to cache and include a beta header ("anthropic-beta": "prompt-caching-2024-07-31") in API calls.
The system checks for cached content on new calls and uses it if available. Caches have a 5-minute lifetime, refreshed with each use. Prompts are cached at 1024-token boundaries.
Cost and Performance Benefits
Up to 90% reduction in API input costs
Up to 85% reduction in latency for long prompts
Initial caching call: 25% more than base input token price
Subsequent cached content use: 10% of base input token price
Applications Prompt caching enables efficient incorporation of entire codebases, use of hundreds of examples for improved task performance, and processing of large documents. It benefits conversational agents, coding assistants, and applications involving extensive document processing or detailed instruction sets. |
|
|
Read more
|
|
|
|
| Discover a new way to run AI inference |
Meet Ori Serverless Kubernetes, an AI cloud service that brings you the best of Serverless and Kubernetes by blending affordability, powerful scalability, and simple management. Their transparent, pay-per-use pricing means you only pay for the resources you consume, helping you optimize GPU costs.
Autoscaling: Adapt your AI infrastructure to inference demand, whilst optimizing costs.
Simplicity of Serverless: They take care of cluster management, load balancing and scaling.
Vanilla Kubernetes: No refactoring or learning curve for Kubernetes users.
Powerful GPUs: Choose between the NVIDIA H100, L40S, L4, or V100S. Get started with $1000 in matched credit when you sign up. |
|
Start integrating today
| partner with us |
|
|
|
TRENDING SIGNALS
| Language Models |
|
⇧ 5843 Likes | |
Language Models |
|
⇧ 2003 Likes | |
Image Generation |
|
⇧ 2160 Likes | |
Video Generation |
|
⇧ 889 Likes | |
AI Software Engineer |
|
⇧ 425 Likes | | |
|
|
|
|
TRENDING REPOS
| Agent | |
☆ 42,812 Stars |
Helps you build software projects with one-line. It generates user stories, requirements, APIs, and code using a multi-agent system simulating a software company. Supports various LLMs and includes a Data Interpreter for analysis tasks and RAG support. |
| Deepfake |
|
| ☆ 24,016 Stars |
Helps you perform real-time face swaps and create deepfake videos using a single image. It uses LLMs to swap faces in images, videos, and live webcam feeds with high-quality results while maintaining ethical safeguards. It supports both GPU and CPU processing. |
| Multimodal LLMs |
|
☆ 10,570 Stars |
Helps you run a GPT-4V level multimodal LLM on your phone. The models take image, video and text as inputs and provide high-quality text outputs. Outperforms GPT-4V, Gemini Pro in single image understanding. Supports real-time video understanding on iPad. Easy deployment via llama.cpp, ollama, vLLM. |
| |
| |
|
|
|
PYTORCH TIP
| |
TorchLens is a Pytorch library that extracts intermediate outputs and metadata from PyTorch neural network models.
What does it do and why it is helpful Accesses model internals during forward pass without code modification.
Extracts layer activations and model structure metadata.
Aids in debugging, behavior analysis, and understanding information flow.
Works with any PyTorch model, supporting tasks like feature visualization and architecture comparison.
Working example with code The example demonstrates TorchLens with a simple recurrent neural network. The model performs four iterations of a linear transformation followed by basic arithmetic operations. TorchLens logs the forward pass, saving all intermediate outputs. Users can then access specific layer outputs, such as the second pass of the first linear layer, as shown in the printed tensor. |
# To install TorchLens, first install graphviz # (required to generate the network visualizations) sudo apt install graphviz
# and then install TorchLens using pip pip install torchlens -q
class SimpleRecurrent(nn.Module): def __init__(self): super().__init__() self.fc = nn.Linear(in_features=5, out_features=5)
def forward(self, x): for r in range(4): x = self.fc(x) x = x + 1 x = x * 2 return x
simple_recurrent = SimpleRecurrent() model_history = tl.log_forward_pass(simple_recurrent, x, layers_to_save='all', vis_opt='rolled') print(model_history['linear_1_1:2'].tensor_contents) # second pass of first linear layer
""" tensor([[-2.9265, 1.3072, -0.4358, -0.9827, -3.0030], [-2.1188, 0.6454, 0.1414, -1.0781, -2.4770], [-1.8614, 1.0404, -0.7678, -0.7843, -2.4568], [-4.4318, 1.4188, -1.6984, -0.3384, -5.2524], [-1.8989, 0.8374, -0.7154, -0.6169, -2.7501], [-3.2618, 1.5131, 1.2455, -1.7886, -1.5585]], grad_fn= 0>) """
| |
Code in google colab
|
|
|
|
Want to promote your company, product, service, or event to 190,000+ AI researchers and engineers?
|
|
Sponsor
|
|
|
|
|