Lune

ACL2024Top-tier venue

Measuring Political Bias in Large Language Models: What Is Said and How It Is Said

Yejin Bang, Delong Chen, Nayeon Lee, Pascale Fung

Abstract

We propose to measure political bias in LLMs by analyzing both the content and style of their generated content regarding political issues. Existing benchmarks and measures focus on gender and racial biases. However, political bias exists in LLMs and can lead to polarization and other harms in downstream applications. In order to provide transparency to users, we advocate that there should be fine-grained and explainable measures of political biases generated by LLMs. Our proposed measure looks at different political issues such as reproductive rights and climate change, at both the content (the substance of the generation) and the style (the lexical polarity) of such bias. We measured the political bias in eleven opensourced LLMs and showed that our proposed framework is easily scalable to other topics and is explainable.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

17 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Framing Political Bias in Multilingual LLMs Across Pakistani Languages cites From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models
  • Bias in the Mirror : Are LLMs opinions robust to their own adversarial attacks cites From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers17

  1. Vision Language Models are BiasedAn Vo, Khai-Nguyen Nguyen, Mohammad Reza Taesiri, Thi Tuong Vy Dang et al.ICLR 2026 · 68 citations
  2. Bias in the Mirror : Are LLMs opinions robust to their own adversarial attacksVirgile Rennard, Christos Xypolopoulos, Michalis VazirgiannisACL 2025 · 8 citations
  3. Framing Political Bias in Multilingual LLMs Across Pakistani LanguagesAfrozah Nadeem, Mark Dras, Usman NaseemACL 2026 · 6 citations
  4. LLMS ON TRIAL: Evaluating Judicial Fairness For Large Language ModelsYiran Hu, Zongyue Xue, Haitao Li, Siyuan Zheng et al.ICLR 2026 · 4 citations
  5. Designing Effective AI Explanations for Misinformation Detection: A Comparative Study of Content, Social, and Combined ExplanationsYeaeun Gong, Yifan Liu, Lanyu Shang, Na Wei et al.CSCW 2025 · 2 citations

Builds on18

  1. Process for Adapting Language Models to Society (PALMS) with Values-Targeted DatasetsIrene Solaiman, Christy DennisonNeurIPS 2021 · 276 citations
  2. Discovering Language Model Behaviors with Model-Written EvaluationsEthan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen et al.ACL Findings 2023 · 195 citations
  3. Generative Echo Chamber? Effect of LLM-Powered Search Systems on Diverse Information SeekingNikhil Sharma, Q. Vera Liao, Ziang XiaoCHI 2024 · 123 citations
  4. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP ModelsShangbin Feng, Chan Young Park, Yuhan Liu, Yulia TsvetkovACL 2023 · 117 citations
  5. Neural Media Bias Detection Using Distant Supervision With BABE - Bias Annotations By ExpertsTimo Spinde, Manuel Plank, Jan-David Krieger, Terry Ruas et al.EMNLP Findings 2021 · 70 citations

Related papers

  1. Assessing Reliability and Political Bias In LLMs' Judgements of Formal and Material Inferences With Partisan ConclusionsReto Gubelmann, Ghassen KarrayACL 2025
  2. Measuring and Mitigating Media Outlet Name Bias in Large Language ModelsSeong-Jin Park, Kang-Min KimEMNLP 2025
  3. PolBiX: Detecting LLMs' Political Bias in Fact-Checking through X-phemismsCharlott Jakob, David Harbecke, Patrick Parschan, Pia Wenzel Neves et al.EMNLP Findings 2025 · 1 citation
  4. Towards Understanding and Mitigating Social Biases in Language ModelsPaul Pu Liang, Chiyu Wu, Louis-Philippe Morency, Ruslan SalakhutdinovICML 2021 · 495 citations
  5. Quantifying Generative Media Bias with a Corpus of Real-world and Generated News ArticlesFilip Trhlík, Pontus StenetorpEMNLP Findings 2024 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines