Lune

ACL Findings2023

Grammar-based Decoding for Improved Compositional Generalization in Semantic Parsing

Jing Zheng, Jyh-Herng Chow, Zhongnan Shen, Peng Xu

Abstract

Sequence-to-sequence (seq2seq) models have achieved great success in semantic parsing tasks, but they tend to struggle on out-of-distribution (OOD) data. Despite recent progress, robust semantic parsing on large-scale tasks that combine challenges from both compositional generalization and natural language variations remains an unsolved issue. To encourage research in this area, this work introduces CUDON, a large-scale dialogue dataset in the Chinese language, specifically created to evaluate the compositional generalization of semantic parsing. The dataset contains about ten thousand multi-turn complex queries, and provides multiple splits with different degrees of train-test distribution divergence. We have investigated improving compositional generalization through grammar-based decoding on this dataset. With specially designed grammars that leverage program schema, we are able to significantly improve the accuracy of seq2seq semantic parsers on OOD splits: a LSTM-based parser using a Context-free Grammar (CFG) achieves over 25% higher accuracy than a standard seq2seq baseline; a parser using Tree-Substitution Grammar (TSG) improves parsing speed by five to seven times over the CFG parser with only a small accuracy loss. The grammar-based LSTM parsers also outperforms BART-and T5-based seq2seq parsers on the OOD splits, despite having less than one tenth of the parameters and no pretraining. We also validated our approach on the SMCalflow-CS dataset, specifically on the zero-shot learning task.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • COGS: A Compositional Generalization Challenge Based on Semantic Interpretation cites Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on14

  1. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  2. Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue DatasetAbhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta et al.AAAI 2020 · 707 citations
  3. Measuring Compositional Generalization: A Comprehensive Method on Realistic DataDaniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman et al.ICLR 2020 · 401 citations
  4. COGS: A Compositional Generalization Challenge Based on Semantic InterpretationNajoung Kim, Tal LinzenEMNLP 2020 · 149 citations
  5. Learning Compositional Rules via Neural Program SynthesisMaxwell I. Nye, Armando Solar-Lezama, Josh Tenenbaum, Brenden M. LakeNeurIPS 2020 · 120 citations

Related papers

  1. Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?Peter Shaw, Ming-Wei Chang, Panupong Pasupat, Kristina ToutanovaACL 2021
  2. Improving Compositional Generalization in Semantic ParsingInbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner et al.EMNLP Findings 2020 · 48 citations
  3. Span-based Semantic Parsing for Compositional GeneralizationJonathan Herzig, Jonathan BerantACL 2021
  4. Compositional Generalization without Trees using Multiset Tagging and Latent PermutationsMatthias Lindemann, Alexander Koller, Ivan TitovACL 2023
  5. DiNeR: A Large Realistic Dataset for Evaluating Compositional GeneralizationChengang Hu, Xiao Liu, Yansong FengEMNLP 2023
Dusk over the sea between two cliffs drawn in fine vertical lines