Skip to main content
Back

Jina AI's 33M Model Handles 8,192 Tokens on a Laptop

Jina AI's compact 33M-parameter embedding model packs 8192-token context into a tiny footprint, using bidirectional ALiBi instead of positional embeddings.

  • jina-embeddings-v2-small-en is a 33M-parameter English embedding model with an 8,192-token context window.
  • Uses bidirectional ALiBi in a custom JinaBERT encoder instead of learned positional embeddings, enabling length extrapolation.
  • Pretrained on C4, then fine-tuned on 400M+ curated sentence pairs and hard negatives.
  • Trained at 512 tokens but extrapolates to 8k+ without retraining, per the technical report.
  • Apache-2.0 licensed, works with sentence-transformers via trust_remote_code=True, 32.7MB F16.
  • Strong for long-document RAG on modest hardware; English-only, may need fine-tuning for specialized domains.

Long-context retrieval usually forces a tradeoff: chunk documents aggressively and lose semantic coherence, or pay for a heavyweight model that eats VRAM. jina-embeddings-v2-small-en is a 33-million parameter English embedding model that handles sequences up to 8,192 tokens while running comfortably on a laptop.

ALiBi: the trick behind the long context

The model is built on JinaBERT, a BERT variant that replaces learned positional embeddings with ALiBi (Attention with Linear Biases). Instead of embedding position as a learned vector, ALiBi applies a linear bias directly to attention scores. Because those biases generalize beyond token counts seen during training, the model extrapolates well past its training length: it was trained on 512-token sequences yet reliably handles 8,192 tokens. At the time of its release, that represented a 16x increase over the maximum sequence length of leading open-source embedding models.

The bidirectional application of ALiBi is the novel piece here. Prior work used ALiBi exclusively in generative, decoder-only models. Jina's team integrated it into an encoder architecture, making it suitable for producing document embeddings rather than generating text.