Lune

ACL Findings2026

MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations

Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, Marina Danilevsky

Abstract

We present MTRAG-UN, a benchmark for exploring open challenges in multi-turn retrieval augmented generation, a popular use of large language models.We release a benchmark of 666 tasks containing over 2,800 conversation turns across 6 domains with accompanying corpora.Our experiments show that retrieval and generation models continue to struggle on conversations with UNanswerable, UNderspecified, and NONstandalone questions and UNclear responses.Our benchmark is available at https://github.com/ IBM/mt-rag-benchmark

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Related papers

  1. MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval BenchmarksJunhao Ruan, Abudukeyumu Abudula, Bei Li, Yongjing Yin et al.ACL 2026
  2. LONG²RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point RecallZehan Qi, Rongwu Xu, Zhijiang Guo, Cunxiang Wang et al.EMNLP Findings 2024 · 6 citations
  3. Benchmarking Large Language Models in Retrieval-Augmented GenerationJiawei Chen, Hongyu Lin, Xianpei Han, Le SunAAAI 2024 · 531 citations
  4. Retrieval-Augmented Machine Translation with Unstructured KnowledgeJiaan Wang, Fandong Meng, Yingxue Zhang, Jie ZhouEMNLP Findings 2025 · 6 citations
  5. Benchmarking Retrieval-Augmented Generation in Multi-Modal ContextsZhenghao Liu, Xingsheng Zhu, Tianshuo Zhou, Xinyi Zhang et al.ACM MM 2025 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines