"Get Their Hands Dirty, Not Mine": On Researcher-Annotator Collaboration and the Agency of Annotators

Annotation quality is often framed as post-hoc cleanup of issues caused by annotators. This position paper discusses how and why this narrative limits the scope of improving data quality. We call to consider annotation as a procedural collaboration, outlining three key points in this direction: (1) An issue can be either annotatoror researcher-oriented, where one party is accountable and the other party may lack ability to fix it; (2) yet, they can co-occur and/or have similar consequences, and any specific problem we observe may be a complicated combination; (3) therefore, we need a new language to capture the nuance and holistically describe the full procedure to resolve these issues. To that end, we propose to study how agency is manifested in annotation and picture how this perspective benefits the community more broadly.