"Feels Feminine to Me": Understanding Perceived Gendered Style through Human Annotations

In NLP, language-gender associations are commonly grounded in the author's gender identity, inferred from their language use. However, this identity-based framing risks reinforcing stereotypes and marginalizing individuals who do not conform to normative language-gender associations. To address this, we operationalize the language-gender association as a perceived gender expression of language, focusing on how such expression is externally interpreted by humans, independent of the author's gender identity. We present the first dataset of its kind: 5,100 human annotations of perceived gendered style-human-written texts rated on a five-point scale from very feminine to very masculine. While perception is inherently subjective, our analysis identifies textual features associated with higher agreement among annotators: formal expressions and lower emotional intensity. Moreover, annotator demographics influence their perception: women annotators are more likely to label texts as feminine, and men and non-binary annotators as masculine. Finally, feature analysis reveals that text's perceived gendered style is shaped by both affective and function words, partially overlapping with known patterns of language variation across gender identities. Our findings lay the groundwork for operationalizing gendered style through human annotation, while also highlighting annotators' subjective judgments as meaningful signals to understand perceptionbased concepts. 1