It's All Relative! A Method to Counter Human Bias in Crowdsourced Stance Detection of News Articles

Using human intelligence to identify news articles' political stances is common in research and practical applications. But human judgement can be biased and prone to errors stemming from the comprehension of tasks and political alignment. This paper proposes a relative rating method based on news articles' stances relative to raters' own stances to avoid comprehension inconsistency and to control for human bias in crowdsourced stance detection of news articles. We also show how to use the relative ratings to construct a measure for raters' stances on a political topic and to identify raters whose ratings are of higher quality than others. We implement our proposed methods in an online experiment that recruits Amazon Mechanical Turk users as raters for news articles on Gun Control. Using the data from the experiment, we find evidence that raters' own stances on Gun Control significantly impact ratings of related news articles, both at the individual levels and at the aggregate levels. We also present evidence that our relative-rating-based stance measure captures more information about raters' actual stances than their self-reported stance does.