Zhipu AI's GLM-4-9B-Chat Beats Llama 3 on Math and Tool Calling
Zhipu AI's 9B open-weight chat model beats Llama-3-8B across reasoning, math, code and long context, with a 128K window and 1M variant.
Takeaways
- GLM-4-9B-Chat beats Llama-3-8B-Instruct on MMLU, MATH, HumanEval, and MT-Bench
- 128K context standard, with a separate 1M-token variant for long documents
- Berkeley Function Calling score of 81.00, essentially tied with gpt-4-turbo
- Native support for 26 languages including strong Chinese and Japanese
- Runs at ~6GB VRAM via Ollama with Q4_K_M quantization
- Custom GLM-4 license, not standard open source, see the technical report
Zhipu AI and Tsinghua University’s Knowledge Engineering Group released GLM-4-9B-Chat in June 2024. The instruction-tuned, open-weight model has resurfaced on Hugging Face’s trending list, renewing interest in its combination of nine billion parameters, a 128K-token context window, tool-call generation, and support for 26 languages. A separate checkpoint extends the context window to one million tokens.
For developers, the model’s strongest case comes from Chinese and English performance, author-reported benchmark gains over Llama 3 8B Instruct, and quantized deployments that fit on a consumer GPU for shorter contexts. The weights use a custom license, so the model is more precisely described as open-weight than open-source.
What fits inside nine billion parameters
GLM-4-9B is the public-weight member of Zhipu’s larger GLM-4 family. The base checkpoint targets completion and fine-tuning, while GLM-4-9B-Chat adds preference alignment, multi-turn conversation, structured function calls, and long-context instruction following. Its technical report traces the broader training program from GLM-130B through GLM-4.
Web search and code execution require external services. The model can select a tool and generate its arguments, while the host application supplies the browser, code sandbox, API client, permissions, and result-handling loop.
| Benchmark | What it measures | GLM-4-9B-Chat | Llama 3 8B Instruct |
|---|---|---|---|
| MMLU | Academic knowledge and reasoning | 72.4 | 68.4 |
| MATH | Competition-level mathematics | 50.6 | 30.0 |
| HumanEval | Python code generation | 71.8 | 62.2 |
| MGSM | Multilingual grade-school mathematics | 65.3 | 54.0 |
Zhipu also reports 75.6 on C-Eval, a Chinese knowledge benchmark, and 8.35 on MT-Bench, which evaluates multi-turn chat. These results come from the model’s release materials; prompt templates, decoding settings, contamination, and evaluator versions can affect comparisons. Teams should reproduce the relevant tasks with their own prompts and output constraints.