ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization

As Large Language Models (LLMs) have become capable of generating long and descriptive code summaries, accurate and reliable evaluation of factual consistency has become a critical challenge. However, previous evaluation methods are primarily designed for short summaries of isolated code snippets. Consequently, they struggle to provide fine-grained evaluation of multi-sentence functionalities and fail to accurately assess dependency context commonly found in real-world code summaries. To address this, we propose ReFEree, a referencefree and fine-grained method for evaluating factual consistency in real-world code summaries. We define factual inconsistency criteria specific to code summaries and evaluate them at the segment level using these criteria along with dependency information. These segment-level results are then aggregated into a fine-grained score. We construct a code summarization benchmark with human-annotated factual consistency labels. The evaluation results demonstrate that ReFEree achieves the highest correlation with human judgment among 13 baselines, improving 15-18% over the previous state-of-the-art. Our code and data are available at https: //github.com/bsy99615/ReFEree.git . Introduction Recent advances in Large Language Models (LLMs), such as GPT-4, have made it feasible to automatically generate long and descriptive code summaries (Achiam et al., 2023; Sun et al., 2024) . LLM-powered assistants such as OpenAI's Codex (Chen et al., 2021), GitHub Copilot (GitHub, 2021), and Anthropic's Claude-Code (Anthropic, 2025) are increasingly integrated into real-world development workflows to assist engineers in understanding and reviewing code. However, when the generated summary does not accurately reflect the code's actual implementation,