Skip to main content
Back

Rock Matrix's EdgeRock Runs 512 Long Conversations on One GPU Without Memory Overflow

A 4B recurrent model matches a Qwen3.5-4B transformer backbone on AIME math while cutting attention KV memory by 99.1% at 256K context.

  • EdgeRock (YANchor-4B) is a 4B recurrent model with O(N) generation and O(1) history memory, on GitHub.
  • Scores 82.9% mean pass@1 on AIME 2024-2026, vs 18.8% for RWKV-7 G1j 13.3B.
  • Retains 96.3% of Qwen3.5-4B's AIME mean while delivering 4.12-6.58x its batched throughput.
  • Three memory paths: Gated DeltaNet summary, sliding-window attention, and a learned anchor bank via GQA.
  • Fixed 48.9 GiB history cache for 512 sequences on one H100, versus projected 1,000+ GiB for Qwen at 64K.
  • Apache-2.0 code, custom weight license, Hugging Face bundle with technical report.

Rock Matrix has released EdgeRock, a 4-billion-parameter multimodal model labeled YANchor-4B in the repository. Built from Qwen3.5-4B, it combines recurrent layers, local attention, and a fixed-capacity learned memory bank. Once those components reach capacity, inference state stops growing with context length, targeting workloads where an expanding key-value cache would exhaust GPU memory.

Rock Matrix reports 82.9% mean pass@1 across AIME 2024 through 2026 and 63.6% on HMMT. Pass@1 measures the share of problems solved by the first generated answer under the evaluation settings. The same release reports higher long-generation throughput and lower memory use than several similarly sized models. These figures come from the project’s own evaluation bundle and have not yet been independently reproduced; hardware, precision, batch size, decoding settings, and optimized kernels can materially affect comparisons.

Project-reported comparison of EdgeRock capability, memory use, and throughput
Rock Matrix’s comparison of YANchor-4B with full-attention and fixed-state models.

Three paths keep history bounded

Qwen3.5-4B already mixes recurrent and global-attention layers. YANchor retains the recurrent layers, replaces global-attention positions with local attention, and adds an independent memory system. Each path handles a different part of the token history:

  • Gated DeltaNet maintains a fixed-size recurrent summary of prior tokens.
  • Sliding-window attention computes detailed interactions among recent tokens.
  • Independent GQA memory learns which records to write, retain, and retrieve over longer distances.

The independent memory bank has separate parameters for writing, admitting, and retrieving records. New entries compete with retained ones through blockwise Top-K selection, which preserves the key-value representations of selected records. Current queries retrieve those anchors through grouped-query attention after the original tokens have left the local window.