Skip to main content
Back

Tsinghua's CPM-Bee Brings Open-Source Bilingual AI to Consumer GPUs

OpenBMB's CPM-Bee is a 10B open-source bilingual base model trained on over a trillion tokens, with commercial use allowed and four size variants from 1B to 10B.

  • Open-source bilingual base model: CPM-Bee is a 10B-parameter Chinese-English model from OpenBMB (Tsinghua NLP Lab), free to use and commercially licensable.
  • Trillion-token training: Pre-trained on over one trillion rigorously curated bilingual tokens, one of the largest open-source training runs for a bilingual model.
  • Top Chinese benchmark performance: Ranked #1 on the ZeroCLUE Chinese evaluation benchmark, with English performance comparable to LLaMA.
  • Four compressed sizes: Available as 1B, 2B, 5B, and 10B variants; the 1B model runs on a GTX 1660 with just 4.1 GB VRAM.
  • Structured JSON interface: Uniquely takes structured JSON as input/output, enabling translation, Q&A, fill-in-the-blank, and classification without prompt engineering.
  • Full toolchain included: Ships with scripts for fine-tuning (BMTrain/OpenDelta), compression (BMCook), and low-resource inference (BMInf); available on HuggingFace and GitHub.

CPM-Bee is a 10-billion-parameter, open-source Chinese-English bilingual base model from OpenBMB, the open-source community co-founded by Tsinghua University's NLP Lab (THUNLP) and ModelBest Inc. It is free to use, commercially licensable, and ships in four compressed sizes that can run on consumer GPUs. If you're building applications that need strong Chinese language understanding alongside solid English performance, this is one of the most capable openly available foundation models in that space.

What it is and where it comes from

CPM-Bee is a fully open-source, commercially-usable Chinese-English bilingual base model with a capacity of ten billion parameters, and it is the second milestone achieved through the training process of CPM-Live. OpenBMB is founded and supported by the Natural Language Processing Laboratory of Tsinghua University (THUNLP) and ModelBest Inc., and the team has a deep research foundation in natural language processing and pretrained models.

CPM-Bee started training on October 13, 2022, and has undergone significant improvements in both foundational capabilities and performance compared to its predecessor CPM-Ant. The CPM (Chinese Pretrained Model) series has a long lineage: it includes the first Chinese large model CPM-1, the efficient CPM-2, and the controllable CPM-3. CPM-Bee is the latest and most capable chapter in that story.

A trillion tokens and a structured twist

Utilizing the Transformer auto-regressive architecture, CPM-Bee has been pre-trained on an extensive corpus of trillion-scale tokens, thereby possessing remarkable foundational capabilities. Training data underwent stringent selection, cleaning, and post-processing to ensure quality, positioning it as one of the highest-volume open-source bilingual models available.

The most distinctive architectural decision is how CPM-Bee handles input and output. Rather than accepting raw free-form text like most base models, the model input and output were designed in a JSON structured format during pre-training, enabling users to complete various tasks by simply adjusting different task fields. This means you express tasks declaratively. A translation task looks like: