ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities
Abstract
Evaluating consistency in Large Language Models (LLMs) is crucial for ensuring reliability, particularly in complex, multi-step interactions between humans and LLMs. Traditional self-consistency methods often miss subtle semantic changes in natural language and functional shifts in code or equations, which can accumulate over multiple transformations. To address this, we propose ConsistencyChecker, a tree-based evaluation framework designed to measure consistency through sequences of reversible transformations, including machine translation tasks and AI-assisted programming tasks. In our proposed framework, nodes represent distinct text states, while edges correspond to pairs of inverse operations. Dynamic and LLM-generated benchmarks ensure a fair assessment of the model's generalization ability and eliminate benchmark leakage. Consistency is quantified based on similarity across different depths of the transformation tree. Experiments on eight models from various families and sizes show that ConsistencyChecker can distinguish the performance of different models. Notably, our consistency scores, computed entirely without using WMT paired data, correlate strongly (r > 0.7) with WMT 2024 auto-ranking, demonstrating the validity of our benchmark-free approach. Our implementation is available at https://github.com/ ulab-uiuc/consistencychecker .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on3
- Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language ModelsHaoran Xu, Young Jin Kim, Amr Sharaf, Hany Hassan AwadallaICLR 2024 · 122 citations
- Latent Space Chain-of-Embedding Enables Output-free LLM Self-EvaluationYiming Wang, Pei Zhang, Baosong Yang, Derek F. Wong et al.ICLR 2025
Related papers
- Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChainMarcus J. Min, Yangruibo Ding, Luca Buratti, Saurabh Pujar et al.ICLR 2024 · 39 citations
- EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence CheckingAnjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen et al.EMNLP 2025
- Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?Zhe Yang, Yichang Zhang, Tianyu Liu, Jian Yang et al.EMNLP 2024 · 2 citations
- ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code SummarizationSuyoung Bae, CheolWon Na, Jaehoon Lee, Yumin Lee et al.ACL 2026 · 1 citation
- ReasonerRank: Redefining Language Model Evaluation with Ground-Truth-Free Ranking FrameworksJiamu Zhang, Jiayi Yuan, Andrew Wen, Hoang Anh Duy Le et al.ACL Findings 2025 · 1 citation
