Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions
Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G. Parker, Munmun De Choudhury
Abstract
Large language models have abilities in creating high-volume humanlike texts and can be used to generate persuasive misinformation. However, the risks remain under-explored. To address the gap, this work frst examined characteristics of AI-generated misinformation (AI-misinfo) compared with human creations, and then evaluated the applicability of existing solutions. We compiled human-created COVID-19 misinformation and abstracted it into narrative prompts for a language model to output AI-misinfo. We found signifcant linguistic diferences within human-AI pairs, and patterns of AImisinfo in enhancing details, communicating uncertainties, drawing conclusions, and simulating personal tones. While existing models remained capable of classifying AI-misinfo, a signifcant performance drop compared to human-misinfo was observed. Results suggested that existing information assessment guidelines had questionable applicability, as AI-misinfo tended to meet criteria in evidence credibility, source transparency, and limitation acknowledgment. We discuss implications for practitioners, researchers, and journalists, as AI can create new challenges to the societal problem of misinformation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
48 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations cites Language Models are Few-Shot Learners
- Exploring Lightweight Interventions at Posting Time to Reduce the Sharing of Misinformation on Social Media cites Fake News on Facebook and Twitter: Investigating How People (Don't) Investigate
- Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions cites Are Deepfakes Concerning? Analyzing Conversations of Deepfakes on Reddit and Exploring Societal Implications
- Can LLM-Generated Misinformation Be Detected? cites "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
Cited by top-tier papers48
- Can LLM-Generated Misinformation Be Detected?Canyu Chen, Kai ShuICLR 2024 · 270 citations
- Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow QuestionsSamia Kabir, David N. Udo-Imeh, Bonan Kou, Tianyi ZhangCHI 2024 · 149 citations
- "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language ModelsXinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen et al.CCS 2024 · 132 citations
- The Dark Side of AI Companionship: A Taxonomy of Harmful Algorithmic Behaviors in Human-AI RelationshipsRenwen Zhang, Han Li, Han Meng, Jinyuan Zhan et al.CHI 2025 · 122 citations
- Synthetic Data Generation with Large Language Models for Text Classification: Potential and LimitationsZhuoyan Li, Hangxiao Zhu, Zhuoran Lu, Ming YinEMNLP 2023 · 102 citations
Builds on13
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Fake News on Facebook and Twitter: Investigating How People (Don't) InvestigateChristine Geeng, Savanna Yee, Franziska RoesnerCHI 2020 · 154 citations
- A Systematic Assessment of Syntactic Generalization in Neural Language ModelsJennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox et al.ACL 2020 · 124 citations
- Exploring Lightweight Interventions at Posting Time to Reduce the Sharing of Misinformation on Social MediaFarnaz Jahanbakhsh, Amy X. Zhang, Adam J. Berinsky, Gordon Pennycook et al.CSCW 2021 · 116 citations
- Are Deepfakes Concerning? Analyzing Conversations of Deepfakes on Reddit and Exploring Societal ImplicationsDilrukshi Gamage, Piyush Ghasiya, Vamshi Krishna Bonagiri, Mark E. Whiting et al.CHI 2022 · 77 citations
Related papers
- Working With AI to Persuade: Examining a Large Language Model's Ability to Generate Pro-Vaccination MessagesElise Karinshak, Sunny Xun Liu, Joon Sung Park, Jeffrey T. HancockCSCW 2023 · 163 citations
- Beyond Accuracy: Experts See AI Fact-Checks as Accurate but Less UsefulChenyan Jia, Apoorva Gondimalla, Angie Zhang, David Joseph Mullings et al.CHI 2026 · 1 citation
- On the Risk of Misinformation Pollution with Large Language ModelsYikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov et al.EMNLP Findings 2023 · 85 citations
- Evaluation of LLM Vulnerabilities to Being Misused for Personalized Disinformation GenerationAneta Zugecova, Dominik Macko, Ivan Srba, Róbert Móro et al.ACL 2025 · 18 citations
- Disinformation Capabilities of Large Language ModelsIvan Vykopal, Matús Pikuliak, Ivan Srba, Róbert Móro et al.ACL 2024
