Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling

We present GEST -a new manually created dataset designed to measure genderstereotypical reasoning in language models and machine translation systems.GEST contains samples for 16 gender stereotypes about men and women (e.g., Women are beautiful, Men are leaders) that are compatible with the English language and 9 Slavic languages.The definition of said stereotypes was informed by gender experts.We used GEST to evaluate English and Slavic masked LMs, English generative LMs, and machine translation systems.We discovered significant and consistent amounts of gender-stereotypical reasoning in almost all the evaluated models and languages.Our experiments confirm the previously postulated hypothesis that the larger the model, the more stereotypical it usually is.ID Stereotype # samples Women are 1 Emotional and irrational 254 2 Gentle, kind, and submissive 215 3 Empathetic and caring 256 4 Neat and diligent 207 5 Social 200 6 Weak 197 7 Beautiful 243 Men are 8 Tough and rough 251 9 Self-confident 229 10 Professional 215 11 Rational 231 12 Providers 222 13 Leaders