Lune

EMNLP Findings2022

Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation

Shivanshu Gupta, Sameer Singh, Matt Gardner

Abstract

A growing body of research has demonstrated the inability of NLP models to generalize compositionally and has tried to alleviate it through specialized architectures, training schemes, and data augmentation, among other approaches. In this work, we study a different approach: training on instances with diverse structures. We propose a model-agnostic algorithm for subsampling such sets of instances from a labeled instance pool with structured outputs. Evaluating on both compositional template splits and traditional IID splits of 5 semantic parsing datasets of varying complexity, we show that structurally diverse training using our algorithm leads to comparable or better generalization than prior algorithms in 9 out of 10 dataset-split type pairs. In general, we find structural diversity to consistently improve sample efficiency compared to random train sets. Moreover, we show that structurally diverse sampling yields comprehensive test sets that are a lot more challenging than IID test sets. Finally, we provide two explanations for improved generalization from diverse train sets: 1) improved coverage of output substructures, and 2) a reduction in spurious correlations between these substructures.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Diverse Demonstrations Improve In-context Compositional Generalization cites Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
  • Diverse Demonstrations Improve In-context Compositional Generalization cites Language Models are Few-Shot Learners
  • Learning to Retrieve Iteratively for In-Context Learning cites Diverse Demonstrations Improve In-context Compositional Generalization
  • Learning to Retrieve Iteratively for In-Context Learning cites Language Models are Few-Shot Learners
  • Language Models are Few-Shot Learners cites Adversarial NLI: A New Benchmark for Natural Language Understanding
  • Language Models are Few-Shot Learners cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Diverse Demonstrations Improve In-context Compositional GeneralizationItay Levy, Ben Bogin, Jonathan BerantACL 2023 · 51 citations
  2. Learning to Retrieve Iteratively for In-Context LearningYunmo Chen, Tongfei Chen, Harsh Jhamtani, Patrick Xia et al.EMNLP 2024 · 2 citations
  3. Understanding Compositional Data Augmentation in Typologically Diverse Morphological InflectionFarhan Samir, Miikka SilfverbergEMNLP 2023

Builds on24

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
  3. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  4. Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations
  5. Measuring Compositional Generalization: A Comprehensive Method on Realistic DataDaniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman et al.ICLR 2020 · 401 citations

Related papers

  1. Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional GeneralizationInbar Oren, Jonathan Herzig, Jonathan BerantEMNLP 2021
  2. Improving Compositional Generalization in Semantic ParsingInbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner et al.EMNLP Findings 2020 · 48 citations
  3. Unobserved Local Structures Make Compositional Generalization HardBen Bogin, Shivanshu Gupta, Jonathan BerantEMNLP 2022 · 15 citations
  4. Evaluating Structural Generalization in Neural Machine TranslationRyoma Kumon, Daiki Matsuoka, Hitomi YanakaACL Findings 2024 · 2 citations
  5. Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?Peter Shaw, Ming-Wei Chang, Panupong Pasupat, Kristina ToutanovaACL 2021
Dusk over the sea between two cliffs drawn in fine vertical lines