Skip to main content
Back

PennyLang Boosts Qwen's Quantum Code Accuracy by 36 Points

A curated dataset of 3,347 PennyLane code samples plus a retrieval pipeline lifts open-source LLM success rates from single digits to over 40%.

  • PennyLang releases 3,347 curated PennyLane code samples as instruction-response pairs.
  • Dataset sourced from GitHub, PennyLane docs, and quantum computing textbooks, cleaned via GPT-4o.
  • With RAG, Qwen 7B jumps from 8.7% to 41.7% success rate on code generation tasks.
  • LLaMa 4 improves from 78.8% to 84.8%; proprietary models see smaller or negative gains.
  • Moderate retrieval budgets (around 75%) beat full-context stuffing by reducing noise.
  • Dataset and evaluation code available at github.com/eBrain4Everyone/Pennylang under CC BY 4.0.

PennyLane occupies a smaller share of public code and documentation than mainstream Python frameworks, leaving general-purpose language models with fewer reliable examples to learn from. The resulting assistants often invent methods or combine valid quantum APIs incorrectly. The team behind PennyLang has released a framework-specific dataset and retrieval pipeline for PennyLane, Xanadu’s open-source platform for differentiable quantum programming and hybrid quantum-classical workflows.

Retrieval gives Qwen a 36-point lift

On the authors’ executable benchmark, retrieval-augmented generation produced the largest gains for models with limited PennyLane knowledge. Success rate measures the percentage of tasks that generated functionally correct code under reference tests.

Reported benchmark results with and without PennyLang retrieval
Model Without RAG With RAG Change
Qwen 7B 8.7% More than 45% More than 36.3 points
LLaMA 4 78.8% 84.8% 6.0 points

The uneven gains show where retrieval earns its cost. Smaller open models can use retrieved examples to compensate for sparse framework knowledge, while models already familiar with PennyLane receive less benefit. Some commercial models recorded minimal or negative gains when given the full retrieved context.

Why quantum code breaks

Quantum programs add constraints that ordinary Python examples rarely cover. Circuits address a fixed set of wires, devices support specific operations, measurements affect valid control flow, and differentiation must follow the framework’s execution model. Common model failures include incorrect decorators, nonexistent gates, incompatible wire counts, unsupported device operations, and invalid measurement sequences.