FairQE: Multi-Agent Framework for Mitigating Gender Bias in Translation Quality Estimation

Quality Estimation (QE) aims to assess machine translation quality without reference translations, but recent studies have shown that existing QE models exhibit systematic gender bias. In particular, they tend to favor masculine realizations in gender-ambiguous contexts and may assign higher scores to gendermisaligned translations even when gender is explicitly specified. To address these issues, we propose FairQE, a multi-agent-based, fairnessaware QE framework that mitigates gender bias in both gender-ambiguous and genderexplicit scenarios. FairQE detects gender cues, generates gender-flipped translation variants, and combines conventional QE scores with LLM-based bias-mitigating reasoning through a dynamic bias-aware aggregation mechanism. This design preserves the strengths of existing QE models while calibrating their genderrelated biases in a plug-and-play manner. Extensive experiments across multiple gender bias evaluation settings demonstrate that FairQE consistently improves gender fairness over strong QE baselines. Moreover, under MQMbased meta-evaluation following the WMT 2023 Metrics Shared Task, FairQE achieves competitive or improved general QE performance. These results show that gender bias in QE can be effectively mitigated without sacrificing evaluation accuracy, enabling fairer and more reliable translation evaluation. * https://platform.openai.com/docs/ models/gpt-4.1-mini * accuracy (addition, mistranslation, omission, untranslated text) * fluency (character encoding, grammar, inconsistency, punctuation, register, spelling) * locale convention (currency, date, name, telephone, time format) * style (awkward) * terminology (inappropriate for context, inconsistent use) * non-translation * other * or no-error -For EACH identified error, assign a severity level: * Critical * Major * Minor Scoring: -Start from a score of 100 points. -Deduct points as follows: * Critical: -15 points * Major: -5 points * Minor: -1 point -The final score must be between 0 and 100.