LLM-box vs. Thinking-box: Designing for Deliberate User Engagement with Distorted Information in Conversational Search

Conversational search, powered by Large Language Models (LLMs), has rapidly become a dominant mode of information seeking. While LLMs reduce the effort of information seeking, they also introduce the risk of distorted information deceptively embedded in responses. Prior work has sought technical mitigations, but such distortion cannot be fully eliminated. We therefore shift the focus to the user level, supporting users in deliberately engaging with information when reading LLM responses. We conducted a user study with frequent conversational search users (N=16), comparing a baseline with two probes—LLM-box (LLM-as-a-judge feedback) and Thinking-box (checkpoints from hallucination patterns)—to examine how these probes influenced users’ recognition of distorted information and their experience of guidance. Our findings indicate that even indirect suggestions significantly improved users’ ability to filter distorted information, while also revealing that guidance must be selective to prevent cognitive overload. These insights point to design implications that enable more deliberate user engagement with LLM responses.