Lune

ACL Findings2021

Confidence-Aware Scheduled Sampling for Neural Machine Translation

Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jie Zhou

Abstract

Scheduled sampling is an effective method to alleviate the exposure bias problem of neural machine translation. It simulates the inference scene by randomly replacing groundtruth target input tokens with predicted ones during training. Despite its success, its critical schedule strategies are merely based on training steps, ignoring the real-time model competence, which limits its potential performance and convergence speed. To address this issue, we propose confidence-aware scheduled sampling. Specifically, we quantify realtime model competence by the confidence of model predictions, based on which we design fine-grained schedule strategies. In this way, the model is exactly exposed to predicted tokens for high-confidence positions and still ground-truth tokens for low-confidence positions. Moreover, we observe vanilla scheduled sampling suffers from degenerating into the original teacher forcing mode since most predicted tokens are the same as ground-truth tokens. Therefore, under the above confidenceaware strategy, we further expose more noisy tokens (e.g., wordy and incorrect word order) instead of predicted ones for high-confidence token positions. We evaluate our approach on the Transformer and conduct experiments on large-scale WMT 2014 English-German, WMT 2014 English-French, and WMT 2019 Chinese-English. Results show that our approach significantly outperforms the Transformer and vanilla scheduled sampling on both translation quality and convergence speed.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation cites Scheduled Sampling Based on Decoding Steps for Neural Machine Translation
  • Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation cites Scheduled Sampling Based on Decoding Steps for Neural Machine Translation
  • Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation cites On the Inference Calibration of Neural Machine Translation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine TranslationSongming Zhang, Yijin Liu, Fandong Meng, Yufeng Chen et al.ACL 2022 · 13 citations
  2. Scheduled Sampling Based on Decoding Steps for Neural Machine TranslationYijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu et al.EMNLP 2021 · 9 citations
  3. Towards Understanding and Improving Knowledge Distillation for Neural Machine TranslationSongming Zhang, Yunlong Liang, Shuaibo Wang, Yufeng Chen et al.ACL 2023 · 8 citations

Builds on6

  1. Uncertainty-Aware Curriculum Learning for Neural Machine TranslationYikai Zhou, Baosong Yang, Derek F. Wong, Yu Wan et al.ACL 2020 · 78 citations
  2. On the Inference Calibration of Neural Machine TranslationShuo Wang, Zhaopeng Tu, Shuming Shi, Yang LiuACL 2020 · 66 citations
  3. Dynamic Data Selection and Weighting for Iterative Back-TranslationZi-Yi Dou, Antonios Anastasopoulos, Graham NeubigEMNLP 2020 · 46 citations
  4. Uncertainty-Aware Semantic Augmentation for Neural Machine TranslationXiangpeng Wei, Heng Yu, Yue Hu, Rongxiang Weng et al.EMNLP 2020 · 20 citations
  5. Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine TranslationWenxiang Jiao, Xing Wang, Shilin He, Irwin King et al.EMNLP 2020 · 18 citations

Related papers

  1. Mixed Cross Entropy Loss for Neural Machine TranslationHaoran Li, Wei LuICML 2021 · 21 citations
  2. Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine TranslationMichalis Korakakis, Andreas VlachosEMNLP Findings 2022
  3. Modeling Fluency and Faithfulness for Diverse Neural Machine TranslationYang Feng, Wanying Xie, Shuhao Gu, Chenze Shao et al.AAAI 2020 · 28 citations
  4. Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine TranslationChulun Zhou, Fandong Meng, Jie Zhou, Min Zhang et al.ACL 2022 · 20 citations
  5. Understanding and Addressing the Under-Translation Problem from the Perspective of Decoding ObjectiveChenze Shao, Fandong Meng, Jiali Zeng, Jie ZhouACL 2024
Dusk over the sea between two cliffs drawn in fine vertical lines