Skip to main content
Back

OrcaRouter Shrinks Flint's 40B Coding Model to Fit on Consumer GPUs

OrcaRouter shipped GGUF builds of an abliterated 40B MoE coding model that run on a 16GB or 32GB GPU with 262K context.

  • OrcaRouter released Flint-abliterated-GGUF, an uncensored 40B MoE with 262K context.
  • Two builds: 12 GB for 16 GB GPUs, 24 GB for 32 GB GPUs, Apache 2.0.
  • Harmful-prompt refusal drops from 86% to 10% with MMLU barely moving.
  • Mix-24G beats Unsloth UD-Q4_K_XL on KLD while being 1.1 GB smaller.
  • Uses measured-KLD knapsack search plus GPTQ-style error-compensated rounding into stock llama.cpp types.
  • Needs llama.cpp master with PR #30928; older Ollama and LM Studio will not load it.

OrcaRouter’s GGUF release packages RecursiveSI’s Flint into two quantized files targeting 16 GB and 32 GB GPUs. The builds also modify the model to reduce refusals, use standard llama.cpp data types, and retain Flint’s roughly 262,000-token context architecture.

GGUF is the model format used by llama.cpp and applications built on it. Quantization compresses weights into lower-precision representations, reducing memory use at the cost of some output fidelity. These builds matter because they make a 39.8-billion-parameter coding model available on a single consumer GPU without custom inference kernels.

Two files, different trade-offs

OrcaRouter’s Flint GGUF builds
Build File size Advertised GPU tier Average expert precision Top-1 match with BF16 Perplexity change
Mix-12G 11.99 GB 16 GB 2.27 bits 88.13% +20.0%
Mix-24G 23.99 GB 32 GB 4.65 bits 96.45% +1.1%

Flint uses a sparse Mixture-of-Experts architecture with 256 routed experts, top-6 routing, and one shared expert. Each token activates about 2.5 billion parameters, reducing computation relative to a dense 39.8B model, while GPU memory must still store the full set of experts.

The attention system combines sliding-window and full attention with learned attention sinks, which provide stable targets over long sequences. OrcaRouter reports a KV cache footprint of about 1.9 GB at 64K tokens and 3.7 GB at 128K tokens. Runtime buffers, the CUDA context, and the KV cache consume memory beyond the GGUF file size, so usable context depends on the GPU, backend, and offload configuration.

A one-vector edit cuts refusal rates

Abliteration identifies a direction in a model’s hidden states associated with refusals, then removes that component from selected weights. OrcaRouter calculated one difference-of-means direction from layer 17 and orthogonalized it out of the attention output projections, routed-expert down projections, and token embeddings. The edit was applied in BF16 before quantization.

Reported effect of the BF16 refusal edit
Metric Original Flint Edited Flint
Harmful-prompt refusal 86% About 10%
MMLU 83.7 83.3
GSM8K 92.0 90.0

Across seven harmful-prompt evaluations, including AdvBench, StrongREJECT, and HarmBench, reported refusal rates fell into the single or low double digits. Residual refusals remain, and quantization can introduce additional behavioral changes. The repository labels the model as a research release and assigns responsibility for its use and outputs to the operator.

The bit budget follows measured damage

Routed experts account for roughly 95% of Flint’s weights, so OrcaRouter concentrated its compression search there. The process used KL divergence, or KLD, to measure how much each candidate precision change altered the model’s output-token distribution. Lower KLD indicates closer agreement with the BF16 reference.

  1. Measure the KLD change from moving each of 85 tensor groups one precision level up or down, using llama.cpp kernels across 64 windows of 2,048 tokens.
  2. Solve a knapsack optimization to select the combination that fits the 12 GB or 24 GB file budget.
  3. Reject proposed moves that fail to improve the measured objective.

After selecting a data type for each tensor, the pipeline applies GPTQ-style error compensation one GGUF block of columns at a time. Each block is quantized with llama.cpp’s implementation, and its rounding error is propagated into later columns using an inverse-Hessian estimate derived from the layer inputs. The resulting files retain stock llama.cpp types and kernels.

Smaller error at each budget

Reported fidelity against the same BF16 reference
Build Size Mean KLD, lower is better Top-1 match, higher is better Perplexity vs. BF16
Unsloth UD-IQ1_M 11.35 GB 0.2203 86.76% +20.4%
Mix-12G 11.99 GB 0.2023 88.13% +20.0%
Unsloth UD-Q4_K_XL 25.05 GB 0.01871 96.30% +1.1%
Mix-24G 23.99 GB 0.01864 96.45% +1.1%

Mix-12G uses about 0.64 GB more than UD-IQ1_M while improving mean KLD and top-1 agreement. Mix-24G is about 1.06 GB smaller than UD-Q4_K_XL, with slightly lower KLD and 0.15 percentage points more next-token agreement. The evaluation data was held out from the precision search, providing a separate check on the allocation method.

These measurements cover token-distribution fidelity. The release provides no corresponding throughput, coding-task, or agent-success results for the quantized files, so the table cannot establish performance on repositories, terminals, or long-running tool workflows.

Use a llama.cpp build with Flint support

A compatible llama.cpp build must include the architecture changes from PR #30928 and chat-template support from PR #30929. Ollama, LM Studio, and other applications embedding earlier llama.cpp revisions may reject the model or mishandle its template, so compatibility depends on the bundled llama.cpp commit.

With the Hugging Face CLI installed, the larger build can be downloaded and launched at a 128K context with:

hf download orcarouter/Flint-abliterated-GGUF Flint-abliterated-Mix-24G.gguf --local-dir ./flint-gguf
llama-server -m flint-gguf/Flint-abliterated-Mix-24G.gguf -c 131072 --jinja

The chat template keeps reasoning enabled. With --jinja, the OpenAI-compatible API places the reasoning trace in message.reasoning_content, the final response in message.content, and tool requests in message.tool_calls. Recommended sampling uses temperature 1.0, top_p 0.95, and top_k 20; the GGUF metadata includes those defaults.

Containment belongs in the runtime

A coding agent with reduced refusal behavior can generate commands, modify files, and invoke tools with fewer model-level safeguards. Runtime controls provide the safety boundary for any tool-enabled deployment:

  • Run generated code and shell commands inside an isolated sandbox with minimal writable mounts.
  • Allowlist tools, network destinations, and filesystem paths.
  • Use short-lived, narrowly scoped credentials kept outside the model context.
  • Validate tool arguments and require approval for destructive or irreversible operations.
  • Scan generated code and protect logs that may contain source, secrets, or reasoning traces.

The 24 GB build carries the case

The release is best suited to air-gapped code assistance, local work on confidential repositories, refusal-mechanism research, red-team evaluations, and experiments with tool-using agents under controlled permissions. Apache 2.0 licensing and stock llama.cpp types simplify local testing and integration.

Task-level evidence remains incomplete because Terminal-Bench, SWE-bench, agentic evaluations, and long-context retrieval tests were not rerun on these quantized files. Mix-12G offers the smaller memory target with a reported 20% perplexity increase over BF16. Mix-24G preserves substantially more of the reference distribution and is the stronger option when 32 GB of VRAM is available.