Technion Builds Causal Graphs That Expose What LLMs Actually Reason
Technion researchers propose a four-phase pipeline that builds causal graphs of LLM reasoning using MCMC-driven counterfactual chains, making model decisions transparent and auditable.
Takeaways
- Technion researchers (with one IBM Research co-author) propose a four-phase pipeline that builds causal graphs of LLM inference from text examples, revealing how models organize concepts to reach predictions.
- The method is model-driven: the target LLM itself discovers discriminative concepts, annotates examples, and generates counterfactual texts used for causal structure learning.
- An MCMC-inspired data expansion step generates chains of counterfactuals to densify the sparse concept space, enabling stable causal discovery via the σ-CG algorithm.
- Tested on Gemini-2-Flash, Qwen3-14B, and GPT-OSS-20B across disease diagnosis, sentiment analysis, and LLM-as-a-judge tasks; graph-based parent sets outperformed all alternative concept subsets in predictive fidelity.
- On structured tasks, models converge on similar concepts; on naturalistic tasks, each model develops distinct latent heuristics with different causal topologies.
- Code is open-sourced under MIT License; full pipeline via Gemini API costs ~$540, open-weight models cost under $5.
Ask an LLM why it made a decision and it will give you an answer. The problem is that answer may have nothing to do with what actually happened inside the model. Chain-of-thought explanations sound plausible but are routinely unfaithful, and attention-based attribution methods only reveal correlations, not causes. A new paper from researchers at the Technion Faculty of Data and Decision Sciences, with a co-author from IBM Research, proposes a fundamentally different approach: instead of asking the model to explain itself, build a causal graph of how it actually reasons.
The work, titled LLM Explainability with Counterfactual Chains and Causal Graphs, introduces a fully automated four-phase pipeline that takes any target LLM and a set of labeled text examples, then outputs a directed graph showing which high-level concepts the model uses, how those concepts relate to each other, and how they combine to produce a final prediction. The paper is authored by Nirit Nussbaum-Hoffer, Nitay Calderon, Liat Ein-Dor, and Roi Reichart.
The problem with existing explainability
Most LLM interpretability research falls into two camps. Mechanistic interpretability digs into neurons, attention heads, and residual stream directions, which is scientifically valuable but rarely produces the kind of explanation a clinician or auditor can act on. Attribution and counterfactual methods are more accessible but remain local and input-centered: they can tell you that a specific phrase influenced a specific prediction, but they do not tell you how the model organizes its reasoning globally across all inputs.
What is missing, the authors argue, is a method that provides a global, causal account of how a model maps human-interpretable concepts to predictions. That is the gap this paper targets.

The four-phase pipeline
The key insight is that the mechanism under study is itself an LLM, which means you can prompt it to generate the data you need for causal analysis. The pipeline exploits this in four steps:
- Label prediction. The target LLM classifies all examples in the dataset. Crucially, the pipeline discards the ground-truth labels and uses the model's own predictions going forward. This ensures the recovered graph reflects the model's perspective, not the dataset's.