Lune

ACL Findings2026

Propaganda Signals in LLMs: Perspectival Divergence and Narrative Framing in the Russia-Ukraine War

Ofir Shabat, Ido Guy, Kira Radinsky

Abstract

Large Language Models (LLMs) are increasingly used to explain, summarize, and translate real-world events, including ongoing geopolitical conflicts. Yet it remains unclear whether they reproduce conflict-specific propaganda and, if so, how this appears in their outputs. We study this question for the Russia-Ukraine war through perspectival divergence, the extent to which model outputs align with competing narratives from different information ecosystems. We construct a conflict-aware evaluation set of neutral English event statements paired with Russian (RU)-and Ukrainian (UA)-oriented reference texts drawn from news outlets and Telegram channels. We then evaluate multiple LLMs under several prompting contexts using a reference-based semantic distance metric that measures directional proximity to RUand UA-oriented references. To explain not only which side a model is closer to but also how that alignment is expressed, we further analyze outputs using five propaganda-relevant categories: Framing & Narrative, Emotional Manipulation, Source & Credibility, Social Pressure & Identity, and Toponymy & Naming. Across models, we find stable, modelspecific leanings and technique profiles that persist across prompts and are not captured by standard factuality-oriented metrics. Our findings show that models that appear neutral under conventional evaluations can still encode systematic, conflict-specific propaganda patterns, underscoring the need for conflict-aware evaluation frameworks.

Ask Lune

Ask anything. The answer quotes the paper.

Lune reads all of it and the papers around it, then shows you the passages behind every answer.

Questions to start from

Lune answers from

the full paper · related work

Get the answer

Free to start. No credit card required.

Builds on7

  1. G-Eval: NLG Evaluation using Gpt-4 with Better Human AlignmentYang Liu, Dan Iter, Yichong Xu, Shuohang Wang et al.EMNLP 2023 · 549 citations
  2. Can Large Language Models Be an Alternative to Human Evaluations?David Cheng-Han Chiang, Hung-yi LeeACL 2023 · 254 citations
  3. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP ModelsShangbin Feng, Chan Young Park, Yuhan Liu, Yulia TsvetkovACL 2023 · 117 citations
  4. Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software EngineeringRuiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan et al.ISSTA 2025 · 27 citations
  5. Measuring Political Bias in Large Language Models: What Is Said and How It Is SaidYejin Bang, Delong Chen, Nayeon Lee, Pascale FungACL 2024 · 21 citations

Related papers

  1. Challenges and Opportunities in Information Manipulation Detection: An Examination of Wartime Russian MediaChan Young Park, Julia Mendelsohn, Anjalie Field, Yulia TsvetkovEMNLP Findings 2022 · 40 citations
  2. Can AI-Generated Persuasion Be Detected? Persuaficial Benchmark and AI vs. Human Linguistic DifferencesArkadiusz Modzelewski, Pawel Golik, Anna Kolos, Giovanni Da San MartinoACL 2026 · 1 citation
  3. NarratEX Dataset: Explaining the Dominant Narratives in News TextsNuno Guimarães, Purificação Silvano, Ricardo Campos, Alípio Mário Jorge et al.EMNLP Findings 2025
  4. PolyNarrative: A Multilingual, Multilabel, Multi-domain Dataset for Narrative Extraction from News ArticlesNikolaos Nikolaidis, Nicolas Stefanovitch, Purificação Silvano, Dimitar Iliyanov Dimitrov et al.ACL 2025
  5. EU DisinfoTest: a Benchmark for Evaluating Language Models' Ability to Detect Disinformation NarrativesWitold Sosnowski, Arkadiusz Modzelewski, Kinga Skorupska, Jahna Otterbacher et al.EMNLP Findings 2024 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines