Arguments that Alter Minds: LLM Rationales Sway Human (and LLM) Notions of Plausibility

We investigate the degree to which human (and LLM) plausibility judgments of multiplechoice commonsense benchmark answers are subject to influence by (im)plausibility arguments for or against an answer, in particular, using rationales generated by LLMs. We collect 3, 000 plausibility judgments from humans and another 13, 600 judgments from LLMs. Overall, we observe increases and decreases in mean human plausibility ratings in the presence of LLM-generated PRO and CON rationales, respectively, suggesting that, on the whole, human judges find these rationales convincing. Experiments with LLMs reveal similar patterns of influence. Our findings demonstrate a novel use of LLMs for studying aspects of human cognition, while also raising practical concerns that, even in domains where humans are "experts" (i.e., common sense), LLMs have the potential to exert considerable influence on people's beliefs. 1 Pro Rationale: A bedroom is a private space where a car-less person can use a radio, smartphone, or other devices to tune into talk radio without external disturbances, ensuring an undisturbed listening experience. Additionally, bedrooms are typically associated with comfort and quiet, reinforcing the ability to focus on the content. Con Rationale: The bedroom is an implausible choice because it is a communal space shared with others in many households, which makes it difficult to ensure full privacy for listening to talk radio. Additionally, a person might not have access to a radio or leisure time in their own bedroom if they share living accommodations. Question: If a car-less person want to listen to talk radio in private, where might they listen to it? Choice: bedroom (gold label) Pro Rationale: A bedroom is a private space … Con Rationale: The bedroom is an implausible choice …