Skip to main content
Back

Taalas AI Chip: 17,000 tokens/sec beats Nvidia by 74x

How Taalas is rewriting the economics of AI inference.

March 1, 2026

Hey,

We’ve spent the last decade simulating intelligence on general-purpose GPUs. Taalas thinks it’s time to cast it directly into silicon. By stripping away programmable cores and external memory, they’ve created a custom AI card for Llama 3-8B that drives inference costs down to a fraction of a cent and pushes speeds to 17,000 tokens per second.

Today, we break down Taalas’s specialized hardware and what it means for the applications you build.

Author photo
Today's Author
Ben Dickson is a veteran software engineer and former CTO with more than a decade of experience analyzing complex AI systems. As a lead contributor to TechCrunch and VentureBeat, he is known as the “Engineer’s Journalist” for translating technical research into practical, production-focused insights.
From Augment Code The coordination layer your AI agent stack is missing
If you're running multiple agents across Claude Code, Codex, or OpenCode, you already know the overhead — context gets lost, branches diverge, and you end up doing the coordination work yourself. Intent fixes that. Every task gets a living spec and a shared workspace. Agents stay aligned. You stay in control.
Sunday Deep Dive
Why Taalas’ 17,000 tokens per second can change the math for AI builders

The AI hardware race now has a new contender. Taalas, a Toronto-based AI hardware startup, recently raised $169 million to create custom silicon that directly embeds artificial intelligence models. According to initial experiments and prototypes, Taalas's processors achieve 17,000 tokens per second on the Llama 3.1 8B model.

The company's architecture abandons traditional programmable accelerators in favor of hardwired hardware. By fixing the neural network directly into the metal, Taalas lowers power consumption considerably compared to current state-of-the-art inference platforms. The improvements hint at a future where AI models can become faster and much more affordable, opening the way for new applications.

What is Taalas and the hardcore approach

Traditional AI data centers rely on graphics processing units (GPUs). GPUs operate on an architecture that physically separates computation cores from memory. When a GPU runs an inference pass on a model, it spends the vast majority of its time and energy moving data from high-bandwidth memory (HBM) to the processor.

HBM involves stacking memory chips vertically to increase data transfer speeds and density, but accessing this off-chip memory remains orders of magnitude slower than accessing on-chip memory. This data movement tax also accounts for a large portion of the power consumed in modern AI data centers, creating a physical bottleneck for running large AI models.

Taalas HC1 chip

Taalas eliminates the memory fetch cycle by unifying storage and computation. Instead of loading weights from external memory, the company translates a specific model's computational graph directly into the physical layout of a chip.

In the company's first-generation HC1 chip, built on a 6-nanometer process featuring 53 billion transistors across an 815 mm² die, the neural network's parameters wire directly into the silicon alongside high-speed SRAM memory.

This architecture removes the need for advanced packaging, 3D memory stacking, or complex liquid cooling. A standard air-cooled server rack can house ten 250W HC1 cards, operating at 12 to 15 kilowatts per rack compared to the 120 to 600 kilowatts required for typical GPU racks.

Performance comparison

Performance and cost benchmarks

The direct-to-silicon approach fundamentally alters unit economics. An Nvidia H200 or B200 GPU serves a user at roughly 230 tokens per second for a Llama 3.1 8B model. Specialized hardware like Cerebras reaches 2,000 tokens per second, while Groq achieves around 600. The Taalas HC1 serves 17,000 tokens per second.

Inference costs follow a similar trajectory. Running Llama 3.1 8B on HC1 costs $0.0075 per million tokens. The same workload on Nvidia GPUs costs between $0.04 and $0.28 per million tokens.

alpha_signal_image_1

According to Taalas, the startup brought its first product to market with a team of 24 people, spending only $30 million of its raised capital to demonstrate that AI models can operate at sub-millisecond speeds.

Taalas expects to deploy a mid-sized reasoning model by spring 2026, followed by a frontier-class large language model on its second-generation HC2 architecture by winter.

The HC2 platform will use multiple chips for logic and memory to support terabyte-scale hard models, moving beyond the single-chip constraints of the current generation.

Why AI builders should care about sub-millisecond latency

Human readers cannot process 17,000 tokens per second, so why does it matter? First, we're seeing an accelerating shift toward tasks that require search and reasoning, where the model generates thousands of chain-of-thought (CoT) tokens before emitting the final answer.

Coding assistants and deep research applications require models to generate tens of thousands of reasoning tokens before outputting a final response. Millisecond latencies enable these systems to operate without breaking a programmer's flow.

On the other hand, low-latency token generation can give a new meaning to "inference-time scaling" laws, enabling developers to generate millions of tokens and explore thousands of solutions in the same amount of time it previously took to reason over an answer.

Unlocking agentic AI at scale

Another area where high token throughput will make a significant change is agentic applications. Rapid token generation makes deploying a fleet of agents economically viable.

Agent-to-agent communication requires models to constantly query, verify, and correct one another. Sub-millisecond speeds remove the latency friction that currently bottlenecks these advanced agentic workflows.

The catch: Flexibility, manufacturing, and obsolescence risks

The ultra-fast speed of token generation isn't without tradeoffs. Hardwiring models into the chip removes flexibility. An HC1 chip built for Llama 3.1 8B cannot run a different model architecture, such as a vision transformer or a convolutional neural network.

The company maintains slight adaptability through configurable context windows and support for low-rank adapters (LoRAs). LoRA is a fine-tuning technique that allows developers to adjust a model's behavior for specific tasks by adding a small set of trainable parameters, rather than retraining the entire neural network. The underlying base model, however, remains permanently fixed in the silicon.

The accuracy tradeoff

Another challenge is the accuracy of the models. Translating models directly to silicon currently requires aggressive quantization, where the precision of the model's parameters are reduced to make it more memory-efficient.

The first generation HC1 chip relies on a custom 3-bit base data type, combining 3-bit and 6-bit parameters. This compression technique reduces memory requirements so the model fits on the chip, but it introduces quality degradation compared to models running on GPUs at higher precision. Taalas plans to adopt standard 4-bit floating-point formats in its next generation to address these accuracy limitations.

Taalas also has to deal with the challenges of a fast-evolving field. Developing custom application-specific integrated circuits (ASICs) traditionally takes two years and tens of millions of dollars.

Taalas has been able to cut this process down to two months, which is quite impressive. But it still has to deal with the difficulties of scaling in the chip industry, which is facing scarce manufacturing capacity and resources all the time.

And yet, even if it manages to manufacture its chips at scale, the rapid two-month manufacturing timeline still exposes buyers to obsolescence risk. AI model architectures evolve quickly, and a hardwired chip may become outdated shortly after deployment if a superior open-weight model releases.

Finally, Taalas will have to fight an uphill battle on the software stack. Most inference engines (e.g., vLLM) and AI optimization libraries (e.g., PagedAttention, FastAttention, etc.) have been primarily designed to work with the Nvidia stack.

While not all of them might apply to Taalas's hardware, the company will still have to provide the tools for developers to adapt their applications to the new tech stack.

Where Taalas fits in the market

So, where do we think Taalas sits in the AI hardware landscape? Hardcore AI processors are suitable for cloud service providers operating at massive scale. Spending millions of dollars on new hardware only makes sense if you can recoup the costs by serving the models at scale.

Another possible sector that might benefit from model-specific hardware are enterprises running highly specialized domain models with long shelf lives. Running the model for several years will justify the upfront cost of manufacturing or purchasing special hardware for it.

Cloud providers also have the resources and talent to abstract away the complexity of setting up the hardware and execution environment, providing an API for developers to have access to the hardware (kind of like Groq).

Taalas is not a sure bet but it surely is an interesting direction to keep tabs on. In a field that is evolving faster than any other technology, wild ideas can turn out to be huge hits.

Together with:
Sponsor Logo
At Alpha Signal, our mission is to build a sharp, engaged community focused on AI, machine learning, and cutting-edge language models, helping over 200,000 developers stay informed and ahead. We’re passionate about curating the best in AI, from top research and trending technical blogs to expert insights and tailored job opportunities. We keep you connected to the breakthroughs and discussions that matter, so you can stay in the loop without endless searching. We also work closely with partners who value the future of AI, including employers and advertisers who want to reach an audience as passionate about AI as we are.

Our partnerships are based on shared values of ethics, responsibility, and a commitment to building a better world through technology.Privacy is a priority at Alpha Signal. Our Privacy Policy clearly explains how we collect, store, and use your personal and non-personal information. By using our website, you accept these terms, which you can review on our website. This policy applies across all Alpha Signal pages, outlining your rights and how to contact us if you want to adjust the use of your information. We’re based in the United States. By using our site, you agree to be governed by U.S. laws.
Looking to promote your company, product, service, or event to 250,000+ AI developers?
Work with us