Skip to main content
Back

Zhipu AI's GLM-4-9B-Chat Beats Llama 3 on Math and Tool Calling

Zhipu AI's 9B open-weight chat model beats Llama-3-8B across reasoning, math, code and long context, with a 128K window and 1M variant.

  • GLM-4-9B-Chat beats Llama-3-8B-Instruct on MMLU, MATH, HumanEval, and MT-Bench
  • 128K context standard, with a separate 1M-token variant for long documents
  • Berkeley Function Calling score of 81.00, essentially tied with gpt-4-turbo
  • Native support for 26 languages including strong Chinese and Japanese
  • Runs at ~6GB VRAM via Ollama with Q4_K_M quantization
  • Custom GLM-4 license, not standard open source, see the technical report

Zhipu AI and Tsinghua University’s Knowledge Engineering Group released GLM-4-9B-Chat in June 2024. The instruction-tuned, open-weight model has resurfaced on Hugging Face’s trending list, renewing interest in its combination of nine billion parameters, a 128K-token context window, tool-call generation, and support for 26 languages. A separate checkpoint extends the context window to one million tokens.

For developers, the model’s strongest case comes from Chinese and English performance, author-reported benchmark gains over Llama 3 8B Instruct, and quantized deployments that fit on a consumer GPU for shorter contexts. The weights use a custom license, so the model is more precisely described as open-weight than open-source.

What fits inside nine billion parameters

GLM-4-9B is the public-weight member of Zhipu’s larger GLM-4 family. The base checkpoint targets completion and fine-tuning, while GLM-4-9B-Chat adds preference alignment, multi-turn conversation, structured function calls, and long-context instruction following. Its technical report traces the broader training program from GLM-130B through GLM-4.

Web search and code execution require external services. The model can select a tool and generate its arguments, while the host application supplies the browser, code sandbox, API client, permissions, and result-handling loop.

Author-reported comparison with Llama 3 8B Instruct
Benchmark What it measures GLM-4-9B-Chat Llama 3 8B Instruct
MMLU Academic knowledge and reasoning 72.4 68.4
MATH Competition-level mathematics 50.6 30.0
HumanEval Python code generation 71.8 62.2
MGSM Multilingual grade-school mathematics 65.3 54.0

Zhipu also reports 75.6 on C-Eval, a Chinese knowledge benchmark, and 8.35 on MT-Bench, which evaluates multi-turn chat. These results come from the model’s release materials; prompt templates, decoding settings, contamination, and evaluator versions can affect comparisons. Teams should reproduce the relevant tasks with their own prompts and output constraints.