Shanghai AI Lab's GraphGen Fixes Hallucinating Training Data With Knowledge Graphs
GraphGen uses knowledge graphs to generate targeted SFT training data, closing knowledge blind spots in LLMs with structured, diverse QA pairs
Takeaways
- GraphGen builds a knowledge graph from source documents, then generates targeted SFT training data that fills the specific knowledge gaps of the model being trained.
- A calibration-based scoring step (ECE) identifies knowledge blind spots in the trainee model, prioritizing rare long-tail facts over common knowledge the model already knows.
- On downstream benchmarks, GraphGen-trained models outperform Qwen2.5-7B-Instruct on GPQA-Diamond (40.0 vs 33.3), AIME25 (22.7 vs 7.2), and SeedBench (65.9 vs 51.5).
- The framework extends beyond SFT: a rephrase pipeline improves pretraining by ~1 point with zero new data, and fill-in-the-blank generation enables RLVR in knowledge-intensive domains.
- Available as pip install graphg, with support for vLLM, SGLang, HuggingFace, Ollama, OpenAI, and scientific databases like NCBI and UniProt.
- The paper was accepted to ACL 2026; a companion RLVR paper was also accepted to the same conference.
GraphGen is an open-source framework from Shanghai AI Lab that takes a fundamentally different approach to synthetic training data generation. Instead of asking an LLM to freely produce question-answer pairs from raw text, it first builds a structured knowledge graph from your source documents, then uses that graph to identify exactly what your target model doesn't know, and only then generates training data to fill those gaps.
The result is SFT data that is factually grounded, covers rare long-tail knowledge, and spans multiple levels of reasoning complexity. The accompanying paper was accepted to ACL 2026, and a follow-up work extending the framework to reinforcement learning was also accepted to the same conference.
The problem with "just ask an LLM"
Most synthetic data pipelines today work by prompting a capable model to generate QA pairs from a document. It sounds reasonable, but it has four compounding failure modes:
- Hallucination: existing approaches frequently suffer from factual inaccuracies, since nothing anchors the generated content to ground truth.
- Long-tail blindness: LLMs are optimized for token prediction, so they gravitate toward common knowledge and systematically skip the rare, domain-specific facts that are often the whole point of fine-tuning.
- Shallow structure: free-form generation produces simple one-hop QA pairs that don't train multi-step reasoning.
- Homogenization: repetitive templates and similar difficulty levels can cause overfitting and, in extreme cases, model collapse.
GraphGen addresses all four by using a knowledge graph as an intermediary. The graph forces factual grounding, its structure enables multi-hop reasoning, and a calibration-based scoring step explicitly prioritizes the knowledge the model is weakest on.
A four-stage pipeline
The GraphGen framework consists of a four-step workflow utilizing two interdependent LLMs: a powerful Synthesizer Model for knowledge extraction and rephrasing, and a Trainee Model, which is the target model for SFT.