DASLab Squeezes a 176.9B Coding Model Down to 32GB Hardware
ISTA's DASLab shrinks a 176.9B-parameter Qwen mixture-of-experts model to 29.6GB resident memory by pruning half its experts and quantizing the rest to 3.5 bits.
Takeaways
- ISTA-DASLab released Qwen3.8-Flash-Next GSQ-RCO Coder, a 176.9B MoE compressed to 58.4GB total, 29.6GB resident.
- Half the 512 experts per layer removed via RCO; remaining weights quantized to 3.5 bits per weight.
- Retains 91.3% of BF16 SWE-bench Verified score and 98.7% of LiveCodeBench v6.
- Calibration mixture targets code, agentic tool use, vision, and spatial reasoning; general capability degrades.
- Runs unmodified in llama.cpp, Ollama, LM Studio, vLLM; Apache-2.0 license inherited from base.
- Reports of 44 tok/s output on 32GB consumer hardware; based on GSQ and RCO papers.
IST Austria’s Deep Algorithms and Systems Lab, or DASLab, has released the Qwen3.8 coder model, a coding-focused compression of Qwen’s Flash-Next mixture-of-experts model. The lab removed half of its routed experts, quantized the remaining weights to 3.5 bits per weight, and packaged part of the model as disk-backed lookup data.
Flash-Next contains 176.9 billion total parameters, but its sparse architecture activates only a small subset for each token. Sparse activation reduces computation while leaving every expert to be stored somewhere. DASLab’s release targets that storage cost, reducing the 354GB BF16 model to a 58.4GB two-shard GGUF package and reporting 29.6GB of model data that must remain memory-resident.
- Original model: 176.9B parameters and 354GB in BF16, a 16-bit weight format
- Expert count: 512 routed experts per layer, reduced to 256
- Active experts: 10 per token
- Quantization: 3.5 bits per surviving weight
- GGUF package: 58.4GB across two shards
- Reported resident data: 29.6GB, with an n-gram lookup shard available from disk
- Reported effective precision: 1.89 bits per original transformer parameter
- License: Apache 2.0, inherited from the base model
The 1.89-bit figure averages weight storage over the original transformer parameter count, including experts removed during pruning. It should not be used to calculate the complete GGUF package size, which also includes metadata and the n-gram lookup data.
Half the experts leave the router
Across 48 layers, Flash-Next contains 24,576 routed experts and selects 10 for each token. DASLab says expert matrices account for 95% of the weights considered during its search, making them the main target for reducing storage.
Pruning deletes complete expert matrices and narrows each layer’s routing pool from 512 experts to 256. Quantization then encodes the retained weights at lower precision. Combining the two methods reduces both the number of stored weights and the space required by each survivor.
RCO chooses the survivors
The RCO paper describes selecting experts by minimizing KL divergence between the original and compressed models on calibration data. KL divergence measures how much the compressed model’s predicted token distribution differs from the original model’s distribution.
Expert usefulness depends on the inputs used during calibration. DASLab emphasized code, agentic tool use, vision, and spatial reasoning, directing the search toward experts that support those workloads. The authors expect broader capabilities to degrade and direct general-purpose users to the general GSQ-RCO model.
GGUF stores one expert count for the entire model, requiring all 48 layers to retain exactly 256 experts. RCO optimizes the loss jointly across the model while enforcing that quota during each layer’s projection step. The method can therefore choose different experts in each layer while preserving the uniform count required by the format.
Coding survives better than agentic work
The project reports the following results under its high-reasoning-effort evaluation setting:
| Benchmark | BF16 | Compressed | Change | Retained |
|---|---|---|---|---|
| SWE-bench Verified | 82.80 | 75.60 | −7.20 | 91.3% |
| LiveCodeBench v6 | 87.43 | 86.28 | −1.15 | 98.7% |
LiveCodeBench measures single-shot code generation, where the compressed model remains close to the BF16 baseline. SWE-bench Verified requires longer agentic trajectories over real repositories, giving routing and generation errors more opportunities to accumulate.
Reproducing these figures requires matching the project’s prompts, reasoning setting, inference stack, and benchmark harness. The reported retention ratios cover these two coding evaluations and do not establish equivalent retention across other tasks or runtime configurations.
Vision survives when calibration includes it
An early RCO search used only code and agentic calibration data, preserving coding scores while severely degrading vision. The optimization had no vision examples against which to value vision-related experts, so those experts offered little benefit to its measured objective.
The released calibration mixture adds vision and spatial-reasoning inputs. Its pelican-on-a-bicycle SVG prompt produces recognizable versions of both objects, providing a qualitative smoke test; the reported results include no numerical vision score.
Run both GGUF shards together
The repository contains two GGUF shards that must remain in the same directory. llama.cpp discovers the second shard when the command points to the first file.
llama.cpp
llama-cli -m Qwen3.8-Flash-Next-GSQ-RCO-Coder-00001-of-00002.gguf \
-p "Refactor this function to be iterative." -ngl 99
The -ngl 99 option requests GPU offload for all available layers. Actual placement depends on available VRAM, runtime buffers, and the selected context length.
Ollama
ollama run hf.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M
The costs behind the 32GB target
- Runtime headroom: The reported 29.6GB covers resident model data. The KV cache, compute buffers, runtime state, and allocator overhead require additional memory.
- Context memory: KV-cache usage grows with context length. Community reports cited by the project estimate about 64GB of combined RAM and VRAM for the full 265K-token context window.
- Disk access: The second shard contains an n-gram lookup table that can remain disk-backed, making storage speed and operating-system page caching relevant to latency.
- Throughput: Community measurements report roughly 44 output tokens per second and 1,300 prefill tokens per second on consumer hardware with 32GB of memory. Prefill measures prompt ingestion. Hardware and context details determine whether those figures transfer to another system.
- Capability coverage: The calibration mixture favors selected workloads, and the larger SWE-bench decline shows that extended agentic tasks remain more sensitive to compression.
Calibration becomes a deployment knob
DASLab, which previously developed GPTQ, now combines GSQ for low-bit weights with RCO for expert selection. Its broader GSQ-RCO family includes non-uniform GGUF quantizations of Qwen3.8-27B reported as small as 11.8GB.
Representative calibration data determines how the surviving expert capacity is allocated. The released profile favors coding, tool use, vision, and spatial reasoning, allowing a model with 176.9B total parameters to target workstation-class hardware. Production suitability depends on matching that profile to real prompts and budgeting memory beyond the 29.6GB resident-model figure.