Automated Testing for Machine Translation via Constituency Invariance

With the development of deep neural networks, machine translation has achieved significant progress and integrated with people’s daily lives to assist in various tasks. However, machine translators, which are essentially one kind of software, also suffer from software defects. Translation errors might cause misunderstanding or even lead to marketing blunders, and political crisis. Thus, almost all translation service providers have feedback channels of incorrect translations to collect training data and improve product performance. Inspired by the syntax structure analysis, we introduce the constituency invariance, which reflects the structural similarity between a simple sentence and sentences derived from it, to test machine translators. We implement it into an automated tool CIT to detect translation errors by checking the constituency invariance relation between the translation results. CIT adopts constituency parse trees to represent the syntactic structures of sentences and employs an efficient data augmentation method to derive multiple new sentences based on one sentence. To validate CIT, we experiment with three widely-used machine translators, i.e., Bing Microsoft Translator, Google Translate, and Youdao Translator. With 600 seed sentences as input, CIT detects 2212, 1910, and 1590 translation errors with around 77% precision. We have submitted detected errors to the development teams. Until we submit this paper, Google, Bing, and Youdao have fixed 15.4%, 32.0%, 14.3% of reported errors, respectively.