ACL Findings2025
LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation
Eunsu Kim, Juyoung Suk, Seungone Kim, Niklas Muennighoff, Dongkwan Kim, Alice Oh
Abstract
We introduce LLM-AS-AN-INTERVIEWER, a novel paradigm for evaluating large language models (LLMs). This approach leverages multi-turn interactions where the LLM interviewer actively provides feedback on responses and poses follow-up questions to the evaluated LLM. At the start of the interview, the LLM interviewer dynamically modifies datasets to generate initial questions, mitigating data contamination. We apply the LLM-as-an-Interviewer framework to evaluate six models on the reasoning, factuality and instruction-following tasks. Our results show that the framework effectively provides insights into LLM performance, including the quality of initial responses, adaptability to feedback, and ability to address follow-up queries like clarification or additional knowledge requests. The framework also addresses key limitations of conventional methods like LLM-as-a-Judge, including verbosity bias and inconsistency across runs. Finally, we propose the Interview Report, which aggregates insights from the interview process, providing examples and a comprehensive analysis of the LLM's strengths and weaknesses. This report offers a detailed snapshot of the model's realworld applicability 12 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge cites BERTScore: Evaluating Text Generation with BERT
- Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation cites Detecting Pretraining Data from Large Language Models
- RefineBench: Evaluating Refinement Capability of Language Models via Checklists cites Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models
- RefineBench: Evaluating Refinement Capability of Language Models via Checklists cites Detecting Pretraining Data from Large Language Models
Cited by top-tier papers5
- From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgeDawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi et al.EMNLP 2025 · 37 citations
- s1: Simple test-time scalingNiklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li et al.EMNLP 2025 · 33 citations
- RefineBench: Evaluating Refinement Capability of Language Models via ChecklistsYoung-Jun Lee, Seungone Kim, Byung-Kwan Lee, Minkyeong Moon et al.ICLR 2026 · 13 citations
- Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic EvaluationSimin Chen, Yiming Chen, Zexin Li, Yifan Jiang et al.EMNLP 2025 · 2 citations
- YIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation AgentsVictor De Lima, Grace Hui YangACL 2026
Builds on15
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceWei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos et al.ICML 2024 · 1,212 citations
- Prometheus: Inducing Fine-Grained Evaluation Capability in Language ModelsSeungone Kim, Jamin Shin, Yejin Choi, Joel Jang et al.ICLR 2024 · 468 citations
- Detecting Pretraining Data from Large Language ModelsWeijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang et al.ICLR 2024 · 365 citations
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language FeedbackXingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen et al.ICLR 2024 · 308 citations
Related papers
- Praetor: A Fine-Grained Generative LLM Evaluator with Instance-Level Customizable Evaluation CriteriaYongqi Leng, Renren Jin, Yue Chen, Zhuowen Han et al.ACL 2025
- StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction FollowingJinnan Li, Jinzhe Li, Yue Wang, Yi Chang et al.ACL Findings 2025 · 3 citations
- LexInstructEval: Lexical Instruction Following Evaluation for Large Language ModelsHuimin Ren, Yan Liang, Baiqiao Su, Chaobo Sun et al.AAAI 2026
- Evaluating Language Models in Realistic Conversational ContextsIlija Subasic, Andrew Rabinovich, Zhao ChenICML 2026
- KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language ModelsZhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang et al.ACL 2024
