Lune

ACL Findings2025

Understand User Opinions of Large Language Models via LLM-Powered In-the-Moment User Experience Interviews

Mengqiao Liu, Tevin Wang, Cassandra A. Cohen, Sarah Li, Chenyan Xiong

Abstract

Which large language model (LLM) is better? Every evaluation tells a story, but what do users really think about current LLMs? This paper presents CLUE, an LLM-powered interviewer that conducts in-the-moment user experience interviews, right after users interact with LLMs, and automatically gathers insights about user opinions from massive interview logs. We conduct a study with thousands of users to understand user opinions on mainstream LLMs, recruiting users to first chat with a target LLM and then be interviewed by CLUE. Our experiments demonstrate that CLUE captures interesting user opinions, e.g., the bipolar views on the displayed reasoning process of DeepSeek-R1 and demands for information freshness and multi-modality. Our code and data are at https://github.com/ cxcscmu/LLM-Interviewer.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on3

  1. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  2. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  3. Justice or Prejudice? Quantifying Biases in LLM-as-a-JudgeJiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen et al.ICLR 2025

Related papers

  1. LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM EvaluationEunsu Kim, Juyoung Suk, Seungone Kim, Niklas Muennighoff et al.ACL Findings 2025 · 4 citations
  2. Collecting Qualitative Data at Scale with Large Language Models: A Case StudyAlejandro Cuevas Villalba, Jennifer V. Scurrell, Eva Maxfield Brown, Jason Entenmann et al.CSCW 2025 · 14 citations
  3. LLM Comparator: Interactive Analysis of Side-by-Side Evaluation of Large Language ModelsMinsuk Kahng, Ian Tenney, Mahima Pushkarna, Michael Xieyang Liu et al.IEEE VIS 2024 · 23 citations
  4. Expectation Alignment of Language Models for Real-World User ExpectationsMiaomiao Li, Yang Wang, Bin Liang, Shudong Liu et al.ICML 2026
  5. DiscoverLLM: From Executing Intents to Discovering ThemTae Soo Kim, Yoonjoo Lee, Jaesang Yu, John Chung et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines