MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies

Autoregressive language models are trained by minimizing the cross-entropy of the model distribution Q θ relative to the data distribution Pthat is, minimizing the forward cross-entropy, which is equivalent to maximum likelihood estimation (MLE). We have observed that models trained in this way may "over-generalize", in the sense that they produce non-human-like text. Moreover, we believe that reverse crossentropy, i.e., the cross-entropy of P relative to Q θ , is a better reflection of how a human would evaluate text generated by a model. Hence, we propose learning with MIXCE, an objective that mixes the forward and reverse crossentropies. We evaluate models trained with this objective on synthetic data settings (where P is known) and real data, and show that the resulting models yield better generated text without complex decoding strategies. https://github.com/bloomberg/ mixce-acl2023 * Work done during an internship at Bloomberg. 1 Unbiased sampling is vanilla random sampling, i.e., sampling with temperature=1.0. It is also called ancestral sampling (Eikema and Aziz, 2020) or pure sampling (Holtzman