Skip to main content
Back

Tsinghua's Vec-LUT Runs Llama 3 8B 4.2x Faster on Plain CPUs

A new vector lookup table design cuts parallel LLM inference bottlenecks on CPUs, delivering up to 4.2x speedup for ternary models on edge devices.

  • Vec-LUT introduces a 1-to-N vector lookup that replaces scalar LUT for parallel LLM inference on CPUs.
  • Up to 4.2x speedup over T-MAC, bitnet.cpp, and llama.cpp across 5 edge devices and 3 ternary models.
  • Two-core CPU inference on Snapdragon 8 Elite beats the device's Hexagon NPU backend by 1.1x.
  • New 1.60 bits-per-weight lossless packing is the most compact ternary format published so far.
  • Serves 273.5 tokens/s on a 50-cent-per-hour AWS Graviton 3 server under continuous batching.
  • Open source at vlut.cpp, integrated into llama.cpp, accepted at MobiSys 2026.

Running large language models directly on phones, laptops, and Raspberry Pis has been quietly transformed by two ideas: squeezing weights down to 1.58 bits (ternary values of -1, 0, 1), and replacing multiplication with table lookups. Together they let plain CPUs beat NPUs at ultra-low-bit inference. There has been a catch, and a group from Tsinghua's Institute for AI Industry Research just published a fix called Vec-LUT that outperforms state-of-the-art baselines by up to 4.2 times on real hardware.

Lookup-table (LUT) inference falls apart the moment you process more than one token at a time. That includes prefilling a prompt, batched serving, speculative decoding, and parallel test-time scaling. LUT-based inference underutilizes memory bandwidth during parallel work, and the root cause is the scalar LUT paradigm, which performs repetitive and non-contiguous memory accesses for each token. Vec-LUT rebuilds the lookup around vectors instead of scalars, and the win shows up everywhere from an Intel i7 to a Snapdragon phone.

Why table lookups replace multiplication

In a ternary model each weight is one of three values, so any small group of weights has only a limited number of possible bit patterns. You can precompute the dot product of every pattern with a slice of the activation vector, dump the results into a table, and then "multiply" by reading the table with the actual weights as indices. No dequantization, no multiplication, just addressed loads and adds. That is why LUT-based design swaps arithmetic for cheap table access, and the vector variant turns random lookup into contiguous vector addition.

The problem hidden in the classic scalar approach is that each token wants its own table. If you are prefilling a 512-token prompt, you build 512 tables, and for each weight index the CPU has to load a different table entry from a different address. Those tables can be hundreds of megabytes; the L1 cache is tens of kilobytes. The result is thrashing, and bandwidth utilization drops below 40 percent.