Skip to main content
Back

Ekka Hunts Down Silent AI Bugs That Crash Accuracy by 30%

Ekka is a new agentic system that automatically pinpoints the root cause of silent quality regressions in LLM serving frameworks like vLLM and SGLang, achieving 80% accuracy at ICML 2026.

Paper, Full Access
  • What: Researchers introduce Ekka, an automated system for diagnosing silent quality regressions in LLM serving frameworks, accepted at ICML 2026.
  • Problem: Silent errors in frameworks like vLLM and SGLang degrade model output quality without any crash or error signal, taking developers months to diagnose manually.
  • How it works: Ekka uses an LLM agent to map semantically equivalent components across a buggy framework and a reference (e.g., HuggingFace), then applies change-point analysis on intermediate tensor activations to pinpoint the divergence.
  • Results: 80% pass@1 and 88% pass@5 diagnosis accuracy on a real-world benchmark, outperforming prior systems by 24-34%, at ~$30 per diagnosis.
  • Real-world validation: Ekka found 4 previously unknown silent errors in vLLM and SGLang, all confirmed by framework developers.
  • Limitation: Requires a correct reference implementation to exist; targets model-stack bugs only; no public code release announced.

Your LLM inference engine is running. No crashes, no error logs, no alerts. But somewhere in the stack, a subtle bug has crept in and the model's benchmark accuracy has quietly dropped 30%. Nobody knows why. This is the silent error problem, and it is one of the most painful failure modes in production LLM serving today. A new research paper accepted at ICML 2026 introduces Ekka, an automated diagnosis system designed to hunt down exactly these kinds of bugs.

The bug that doesn't crash anything

LLM serving frameworks are quickly evolving with a complex software stack and a vast number of optimizations, and the rapid development process can introduce silent errors where output quality silently degrades without any explicit error signals. Unlike a segfault or an assertion failure, a silent error lets the engine keep running and returning responses, while the outputs are subtly wrong.

The paper illustrates this with a real vLLM bug: a recent silent error in vLLM caused the Gemma 3 model's accuracy on the Hellaswag benchmark to drop nearly 30% without triggering any runtime errors or warnings. The root cause turned out to be a misuse of sliding window attention buried deep in the model stack, and developers spent months misdiagnosing the issue before identifying it. That kind of debugging timeline is unsustainable as serving frameworks ship updates at an ever-faster pace.

Diagnosing silent errors is notoriously difficult due to the substantial semantic gap between the high-level symptoms and the low-level root causes. The authors studied 90 real-world silent errors from vLLM and SGLang to understand exactly where these bugs come from and how developers currently deal with them.

Pie chart showing root cause distribution of silent errors across framework implementation, model implementation, kernel backend, and numerical precision

Where the bugs actually live

The study's breakdown of root causes is illuminating. Framework implementation issues constitute the largest category (30.6%), involving logic errors within the serving engine itself, such as async engine implementation and CUDA graph compilation. The model implementation category (25.5%) covers errors in defining model architectures or misconfiguring models. Kernel backend bugs (24.5%) arise within specialized compute kernels such as FlashAttention. Rounding out the picture, around 19.4% of issues are related to numerical precision, stemming purely from floating-point instability such as BF16 accumulation errors in the absence of logical defects.