Lune

AAAI2026Top-tier venue

Interactive Evaluation of Large Language Models for Multi-Requirement Software Engineering Tasks

Dimitrios Rontogiannis, Maxime Peyrard, Nicolas Mario Baldwin, Martin Josifoski, Robert West, Dimitrios Gunopulos

Abstract

Standard single-turn, static benchmarks fall short in evaluating the nuanced capabilities of Large Language Models (LLMs) on complex tasks such as software engineering. In this work, we propose a novel interactive evaluation framework that assesses LLMs on multi-requirement programming tasks through structured, feedback-driven dialogue. Each task is modeled as a requirement dependency graph, and an "interviewer" LLM, aware of the ground-truth solution, provides minimal, targeted hints to an "interviewee" model to help correct errors and fulfill target constraints. This dynamic protocol enables fine-grained diagnostic insights into model behavior, uncovering strengths and systematic weaknesses that static benchmarks fail to measure. We build on DevAI, a benchmark of 55 curated programming tasks, by adding ground-truth solutions and evaluating the relevance and utility of interviewer hints through expert annotation. Our results highlight the importance of dynamic evaluation in advancing the development of collaborative code-generating agents.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks cites Adaptive Testing and Debugging of NLP Models
  • IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering cites MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on7

  1. SWE-bench: Can Language Models Resolve Real-world Github Issues?Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao et al.ICLR 2024 · 2,082 citations
  2. MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language FeedbackXingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen et al.ICLR 2024 · 308 citations
  3. Adaptive Testing and Debugging of NLP ModelsMarco Túlio Ribeiro, Scott M. LundbergACL 2022 · 99 citations
  4. DyVal: Dynamic Evaluation of Large Language Models for Reasoning TasksKaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong et al.ICLR 2024 · 92 citations
  5. IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question AnsweringRuosen Li, Ruochen Li, Barry Wang, Xinya DuNeurIPS 2024 · 26 citations

Related papers

  1. When Benchmarks Talk: Re-Evaluating Code LLMs with Interactive FeedbackJane Pan, Ryan Shar, Jacob Pfau, Ameet Talwalkar et al.ACL Findings 2025
  2. JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM EvaluationZhichao Shi, Xuhui Jiang, Chengjin Xu, Cangli Yao et al.ACL Findings 2026
  3. INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and HelpfulnessHung Le, Doyen Sahoo, Yingbo Zhou, Caiming Xiong et al.NeurIPS 2024 · 12 citations
  4. Talk2Code: A Multi-Turn Interaction Benchmark with Dual-Track Evaluation for Code GenerationWeibin Yang, Liangru Xie, Jieyun Cai, Yuxiang Yan et al.AAAI 2026
  5. Agent-as-a-Judge: Evaluate Agents with AgentsMingchen Zhuge, Changsheng Zhao, Dylan R. Ashley, Wenyi Wang et al.ICML 2025
Dusk over the sea between two cliffs drawn in fine vertical lines