Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models

Prompt-based language models like GPT4 and LLaMa have been used for a wide variety of use cases such as simulating agents, searching for information, or for content analysis. For all of these applications and others, political biases in these models can affect their performance. Several researchers have attempted to study political bias in language models using evaluation suites based on surveys, such as the Political Compass Test (PCT), often finding a particular leaning favored by these models. However, there is some variation in the exact prompting techniques, leading to diverging findings, and most research relies on constrained-answer settings to extract model responses. Moreover, the Political Compass Test is not a scientifically valid survey instrument. In this work, we contribute a political bias measured informed by political science theory, building on survey design principles to test a wide variety of input prompts, while taking into account prompt sensitivity. We then prompt 11 different open and commercial models, differentiating between instruction-tuned and non-instructiontuned models, and automatically classify their political stances from 88,110 responses. Leveraging this dataset, we compute political bias profiles across different prompt variations and find that while PCT exaggerates bias in certain models like GPT3.5, measures of political bias are often unstable, but generally more leftleaning for instruction-tuned models. Code and data are available on GitHub 1 . * Corresponding Author 1 https://github.com/MaFa211/theory_grounded_pol_bias Motoki et al. (2024) no no no no no Rozado (2023) no no no no no Rutinowski et al. (2024) no no no no no Fujimoto and Takemoto (2023) no no no no no Rozado (2024) no yes no no no Hartmann et al. (2023) no yes no no yes (voting advice) Thapa et al. (2023) yes no no yes no Feng et al. (2023) yes yes no yes yes (labeling hate speech and misinformation) España-Bonet (2023) no no no no yes (media bias) Ghafouri et al. (2023) yes no no no yes (debate questions) Röttger et al. (2024) yes yes no yes no Wright et al. (2024) yes yes no yes no Ceron et al. (2024)