Skip to main content
Back

E2LLM Cuts Edge AI Waiting Time in Half on Messy Real-World Hardware

E2LLM uses genetic algorithms and dynamic programming to run full LLMs across heterogeneous edge devices, cutting wait times by over 50% versus Splitwise

Paper, Full Access
  • E2LLM is a new framework for serving LLMs across heterogeneous edge/fog devices that cannot individually host a full model.
  • It replicates the full model across multiple device groups, assigning each replica a dedicated PREFILL or DECODE role to exploit their computational differences.
  • A Genetic Algorithm forms optimal device clusters, while Dynamic Programming determines the best layer-partition within each cluster.
  • Compared to the Splitwise baseline, E2LLM cuts average waiting time by over 50% under high-demand conditions.
  • The system handles variable workloads with fluctuating input/output token lengths robustly, unlike static partitioning schemes.
  • No code is publicly released yet; the paper is available at arxiv.org/abs/2606.03770.

Running a large language model on a single powerful GPU is the comfortable default. But what happens when you need inference at the edge -- on a hospital floor, a factory line, or a smart city node -- where no single device has enough memory to hold the model? A new paper introduces E2LLM, a serving framework that tackles this exact problem by intelligently distributing LLMs across fleets of heterogeneous, resource-constrained edge and fog devices.

The Two-Phase Problem Nobody Talks About at the Edge

To understand why this is hard, you need to understand how LLM inference actually works. It has two phases: prefill (processing input tokens in parallel to build the KV cache) and decode (generating output tokens one at a time). These two phases have fundamentally different computational personalities.

In prefill, the model processes all prompt tokens in parallel to generate the first output token and is typically compute-bound due to large matrix multiplications. In decode, it generates subsequent tokens sequentially, becoming memory-bandwidth-bound because of smaller matrix multiplications and frequent parameter and KV-cache accesses. Forcing both phases onto the same hardware is an inherent mismatch -- and on edge devices with tight memory and compute budgets, that mismatch is even more painful.

When prefill and decode are co-located on the same GPUs, prefill requests can monopolize GPU execution and delay decode steps, directly inflating both time-to-first-token (TTFT) and time-per-output-token (TPOT). Cloud-scale systems like Splitwise and DistServe have addressed this by disaggregating the two phases onto separate, high-end GPU clusters. But those solutions assume homogeneous, well-provisioned hardware -- an assumption that breaks down completely at the edge.

Diagram showing the prefill and decode phases of autoregressive LLM inference with KV cache

Edge and Fog: A Different Beast

The core tension is that LLMs are massive and computationally intensive, while edge devices are resource-constrained in terms of power, memory, and processing capabilities. LLMs based on the Transformer architecture are inherently unsuitable for edge computing in their original, full-sized form.

Edge resources differ from cloud resources in several ways: they are resource constrained, meaning they have fewer computational capabilities due to edge devices' smaller processors and lower power budgets; they are heterogeneous, meaning that different processors use different configurations; and their workloads adjustment and applications fight for them. A fog layer sits between these edge nodes and the cloud -- bringing the substantial computational capabilities of central instances closer to consumers in a distributed fashion, lowering latency and optimizing network resources.