Navigating the Political Compass: Evaluating Multilingual LLMs across Languages and Nationalities

Large Language Models (LLMs) have become ubiquitous in today's technological landscape, boasting a plethora of applications and even endangering human jobs in complex and creative fields. One such field is journalism: LLMs are being used for summarization, generation, and even fact-checking. However, in today's political landscape, LLMs could accentuate tensions if they exhibit political bias. In this work, we evaluate the political bias of the 15 most-used multilingual LLMs via the Political Compass Test. We test different scenarios, where we vary the language of the prompt while also assigning a nationality to the model. We evaluate models on the 50 most populous countries and their official languages. Our results indicate that language has a strong influence on the political ideology displayed by a model. In addition, smaller models tend to display a more stable political ideology, i.e. ideology that is less affected by variations in the prompt.

as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301-8327,