ACL Findings2026
Propaganda Signals in LLMs: Perspectival Divergence and Narrative Framing in the Russia-Ukraine War
Ofir Shabat, Ido Guy, Kira Radinsky
摘要
Large Language Models (LLMs) are increasingly used to explain, summarize, and translate real-world events, including ongoing geopolitical conflicts. Yet it remains unclear whether they reproduce conflict-specific propaganda and, if so, how this appears in their outputs. We study this question for the Russia-Ukraine war through perspectival divergence, the extent to which model outputs align with competing narratives from different information ecosystems. We construct a conflict-aware evaluation set of neutral English event statements paired with Russian (RU)-and Ukrainian (UA)-oriented reference texts drawn from news outlets and Telegram channels. We then evaluate multiple LLMs under several prompting contexts using a reference-based semantic distance metric that measures directional proximity to RUand UA-oriented references. To explain not only which side a model is closer to but also how that alignment is expressed, we further analyze outputs using five propaganda-relevant categories: Framing & Narrative, Emotional Manipulation, Source & Credibility, Social Pressure & Identity, and Toponymy & Naming. Across models, we find stable, modelspecific leanings and technique profiles that persist across prompts and are not captured by standard factuality-oriented metrics. Our findings show that models that appear neutral under conventional evaluations can still encode systematic, conflict-specific propaganda patterns, underscoring the need for conflict-aware evaluation frameworks.
问 Lune
尽管问,回答会引用原文。
Lune 会读完全文和相关的 Paper,每个回答都附上所依据的原文段落。
它引用的 Paper7
- G-Eval: NLG Evaluation using Gpt-4 with Better Human AlignmentYang Liu, Dan Iter, Yichong Xu, Shuohang Wang 等EMNLP 2023 · 549 次引用
- Can Large Language Models Be an Alternative to Human Evaluations?David Cheng-Han Chiang, Hung-yi LeeACL 2023 · 254 次引用
- From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP ModelsShangbin Feng, Chan Young Park, Yuhan Liu, Yulia TsvetkovACL 2023 · 117 次引用
- Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software EngineeringRuiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan 等ISSTA 2025 · 27 次引用
- Measuring Political Bias in Large Language Models: What Is Said and How It Is SaidYejin Bang, Delong Chen, Nayeon Lee, Pascale FungACL 2024 · 21 次引用
相关 Paper
- Challenges and Opportunities in Information Manipulation Detection: An Examination of Wartime Russian MediaChan Young Park, Julia Mendelsohn, Anjalie Field, Yulia TsvetkovEMNLP Findings 2022 · 40 次引用
- Can AI-Generated Persuasion Be Detected? Persuaficial Benchmark and AI vs. Human Linguistic DifferencesArkadiusz Modzelewski, Pawel Golik, Anna Kolos, Giovanni Da San MartinoACL 2026 · 1 次引用
- NarratEX Dataset: Explaining the Dominant Narratives in News TextsNuno Guimarães, Purificação Silvano, Ricardo Campos, Alípio Mário Jorge 等EMNLP Findings 2025
- PolyNarrative: A Multilingual, Multilabel, Multi-domain Dataset for Narrative Extraction from News ArticlesNikolaos Nikolaidis, Nicolas Stefanovitch, Purificação Silvano, Dimitar Iliyanov Dimitrov 等ACL 2025
- EU DisinfoTest: a Benchmark for Evaluating Language Models' Ability to Detect Disinformation NarrativesWitold Sosnowski, Arkadiusz Modzelewski, Kinga Skorupska, Jahna Otterbacher 等EMNLP Findings 2024 · 3 次引用
