Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?

Sequence-to-sequence models excel at handling natural language variation, but have been shown to struggle with out-of-distribution compositional generalization. This has motivated new specialized architectures with stronger compositional biases, but most of these approaches have only been evaluated on synthetically-generated datasets, which are not representative of natural language variation. In this work we ask: can we develop a semantic parsing approach that handles both natural language variation and compositional generalization? To better assess this capability, we propose new train and test splits of non-synthetic datasets. We demonstrate that strong existing approaches do not perform well across a broad set of evaluations. We also propose NQG-T5, a hybrid model that combines a highprecision grammar-based approach with a pretrained sequence-to-sequence model. It outperforms existing approaches across several compositional generalization challenges on nonsynthetic data, while also being competitive with the state-of-the-art on standard evaluations. While still far from solving this problem, our study highlights the importance of diverse evaluations and the open challenge of handling both compositional generalization and natural language variation in semantic parsing. * ⇒ x, y for every x, y ∈ D. The search completes when no rule that decreases L(R) can be identified.

To describe the implementation, first let us define several operations over rules and sets of rules. We define the set of rules that can be derived from a given set of rules, R:

We define an operation SPLIT that generates possible choices for splitting a rule into 2 rules: