A Comparison between Pre-training and Large-scale Back-translation for Neural Machine Translation

BERT has been studied as a promising technique to improve NMT. Given that BERT is based on the similar Transformer architecture to NMT and the current datasets for most MT tasks are rather large, how pre-training has managed to outperform standard Transformer NMT models is underestimated. We compare MT engines trained with pre-trained BERT and back-translation with incrementally larger amounts of data, implementing the two most widely-used monolingual paradigms. We analyze their strengths and weaknesses based on both standard automatic metrics and intrinsic test suites that comprise a large range of linguistic phenomena. Primarily, we find that 1) BERT has limited advantages compared with large-scale back-translation in accuracy and consistency on morphology and syntax; 2) BERT can boost the Transformer baseline in semantic and pragmatic tasks which involve intensive understanding; 3) pre-training on huge datasets may introduce inductive social bias thus affects translation fairness.