ACL Findings2023
Pre-Trained Language-Meaning Models for Multilingual Parsing and Generation
Abstract
Pre-trained language models (PLMs) have achieved great success in NLP and have recently been used for tasks in computational semantics. However, these tasks do not fully benefit from PLMs since meaning representations are not explicitly included in the pretraining stage. We introduce multilingual pretrained language-meaning models based on Discourse Representation Structures (DRSs), including meaning representations besides natural language texts in the same model, and design a new strategy to reduce the gap between the pre-training and fine-tuning objectives. Since DRSs are language neutral, crosslingual transfer learning is adopted to further improve the performance of non-English tasks. Automatic evaluation results show that our approach achieves the best performance on both the multilingual DRS parsing and DRS-to-text generation tasks. Correlation analysis between automatic metrics and human judgements on the generation task further validates the effectiveness of our model. Human inspection reveals that out-of-vocabulary tokens are the main cause of erroneous results.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MVP: Multi-task Supervised Pre-training for Natural Language Generation cites Rethinking Supervised Pre-Training for Better Downstream Transferring
- Scope-enhanced Compositional Semantic Parsing for DRT cites Character-level Representations Improve DRS-based Semantic Parsing Even in the Age of BERT
Cited by top-tier papers1
Builds on8
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training DataShuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu et al.ACL 2022 · 115 citations
- Rethinking Supervised Pre-Training for Better Downstream TransferringYutong Feng, Jianwen Jiang, Mingqian Tang, Rong Jin et al.ICLR 2022 · 51 citations
- MVP: Multi-task Supervised Pre-training for Natural Language GenerationTianyi Tang, Junyi Li, Wayne Xin Zhao, Ji-Rong WenACL Findings 2023 · 23 citations
- Character-level Representations Improve DRS-based Semantic Parsing Even in the Age of BERTRik van Noord, Antonio Toral, Johan BosEMNLP 2020 · 22 citations
Related papers
- XLPT-AMR: Cross-Lingual Pre-Training via Multi-Task Learning for Zero-Shot AMR Parsing and Text GenerationDongqin Xu, Junhui Li, Muhua Zhu, Min Zhang et al.ACL 2021
- Graph Pre-training for AMR Parsing and GenerationXuefeng Bai, Yulong Chen, Yue ZhangACL 2022
- Multilingual Pre-training with Universal Dependency LearningKailai Sun, Zuchao Li, Hai ZhaoNeurIPS 2021 · 11 citations
- Improving AMR Parsing with Sequence-to-Sequence Pre-trainingDongqin Xu, Junhui Li, Muhua Zhu, Min Zhang et al.EMNLP 2020 · 57 citations
- Struct-XLM: A Structure Discovery Multilingual Language Model for Enhancing Cross-lingual Transfer through Reinforcement LearningLinjuan Wu, Weiming LuEMNLP 2023
