Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Users

Deep Research (DR) systems help researchers cope with ballooning publishing counts. Such tools synthesize scientific papers to answer research queries, but lack understanding of their users. We address this with MYSCHOLARQA (MYSQA), a personalized DR agent that: 1) infers a profile with a user's research interests; 2) proposes personalized actions for a user's input query; and 3) writes a multi-section report for the query that follows user-approved actions. We first test MYSQA with NLP's standard protocol: we build a benchmark with synthetic users and LLM judges, where MYSQA beats baselines in citation metrics and personalized action-following. However, we suspect this process does not cover all aspects of personalized DR users value, so we interview users in an online version of MYSQA to unmask them. We reveal nine nuanced errors of personalized DR undetectable by our LLM judges, and we study qualitative feedback to form lessons for future DR design. In all, we argue for a pillar of personalization that easy-to-use LLM judges can lead NLP to overlook: real progress in personalization is only possible with real users. 1 When Deep Research Gets to Know You Scholars increasingly turn to LLMs to support their scientific research (Liao et al., 2024) , such as to learn new concepts (August et al., 2023) or brainstorm ideas (Pu et al., 2025) . With publishing rates skyrocketing and literature becoming daunting to track (Parolo et al., 2015) , a new use case of LLMs emerges: Deep Research (DR) tools that answer researchers' queries by retrieving, organizing, and synthesizing papers into multi-section, attributed reports (Asai et al., 2024; Huang et al., 2025) . 1) Infer a Profile ( §2.1 ) 2) Propose Actions ( §2.2 ) 3) Synthesize a Report ( §2.3 ) Researcher Profile Knowledge: -You have expertise on training building scientific QA… [1, 3] Research Style: -You prefer to study models using black-box analysis… [2, 3] [1] [2] [3] Researcher-picked papers Query How can I build the first personalized Deep Research System? List of Actions Research Ideas: -Point out current QA limitations -Give design deployment steps Content: -Focus on scientific QA papers -Discuss deep learning findings