ACL Findings2021
Diagnosing Transformers in Task-Oriented Semantic Parsing
Shrey Desai, Ahmed Aly
Abstract
Modern task-oriented semantic parsing approaches typically use seq2seq transformers to map textual utterances to semantic frames comprised of intents and slots. While these models are empirically strong, their specific strengths and weaknesses have largely remained unexplored. In this work, we study BART (Lewis et al., 2020) and XLM-R (Conneau et al., 2020), two state-of-the-art parsers, across both monolingual and multilingual settings. Our experiments yield several key results: transformer-based parsers struggle not only with disambiguating intents/slots, but surprisingly also with producing syntacticallyvalid frames. Though pre-training imbues transformers with syntactic inductive biases, we find the ambiguity of copying utterance spans into frames often leads to tree invalidity, indicating span extraction is a major bottleneck for current parsers. However, as a silver lining, we show transformer-based parsers give sufficient indicators for whether a frame is likely to be correct or incorrect, making them easier to deploy in production settings.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing cites Conversational Semantic Parsing
Builds on5
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic ParsingXilun Chen, Asish Ghoshal, Yashar Mehdad, Luke Zettlemoyer et al.EMNLP 2020 · 66 citations
- Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection in Task Oriented DialogVarun Gangal, Abhinav Arora, Arash Einolghozati, Sonal GuptaAAAI 2020 · 59 citations
- Conversational Semantic ParsingArmen Aghajanyan, Jean Maillard, Akshat Shrivastava, Keith Diedrick et al.EMNLP 2020 · 3 citations
Related papers
- Improving AMR Parsing with Sequence-to-Sequence Pre-trainingDongqin Xu, Junhui Li, Muhua Zhu, Min Zhang et al.EMNLP 2020 · 57 citations
- Structural Guidance for Transformer Language ModelsPeng Qian, Tahira Naseem, Roger Levy, Ramón Fernandez AstudilloACL 2021
- XSemPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning RepresentationsYusen Zhang, Jun Wang, Zhiguo Wang, Rui ZhangACL 2023 · 4 citations
- Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic ParsingAkshat Shrivastava, Pierce Chuang, Arun Babu, Shrey Desai et al.EMNLP Findings 2021 · 23 citations
- Multilingual Pre-training with Universal Dependency LearningKailai Sun, Zuchao Li, Hai ZhaoNeurIPS 2021 · 11 citations
