Lune

EMNLP2021Top-tier venue

Building Adaptive Acceptability Classifiers for Neural NLG

Soumya Batra, Shashank Jain, Peyman Heidari, Ankit Arun, Catharine Youngs, Xintong Li, Pinar Donmez, Shawn Mei, Shiunzu Kuo, Vikas Bhardwaj, Anuj Kumar, Michael White

Abstract

We propose a novel framework to train models to classify acceptability of responses generated by natural language generation (NLG) models, improving upon existing sentence transformation and model-based approaches. An NLG response is considered acceptable if it is both semantically correct and grammatical. We don't make use of any human references making the classifiers suitable for runtime deployment. Training data for the classifiers is obtained using a 2-stage approach of first generating synthetic data using a combination of existing and new model-based approaches followed by a novel validation framework to filter and sort the synthetic data into acceptable and unacceptable classes. Our 2stage approach adapts to a wide range of data representations and does not require additional data beyond what the NLG models are trained on. It is also independent of the underlying NLG model architecture, and is able to generate more realistic samples close to the distribution of the NLG model-generated responses. We present results on 5 datasets (WebNLG, Cleaned E2E, ViGGO, Alarm, and Weather) with varying data representations. We compare our framework with existing techniques that involve synthetic data generation using simple sentence transformations and/or modelbased techniques, and show that building acceptability classifiers using data that resembles the generation model outputs followed by a validation framework outperforms the existing techniques, achieving state-of-the-art results. We also show that our techniques can be used in few-shot settings using self-training.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs cites RuCoLA: Russian Corpus of Linguistic Acceptability
  • QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgments cites RuCoLA: Russian Corpus of Linguistic Acceptability
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. RuCoLA: Russian Corpus of Linguistic AcceptabilityVladislav Mikhailov, Tatiana Shamardina, Max Ryabinin, Alena Pestova et al.EMNLP 2022 · 19 citations
  2. RuBLiMP: Russian Benchmark of Linguistic Minimal PairsEkaterina Taktasheva, Maxim Bazhukov, Kirill Koncha, Alena Fenogenova et al.EMNLP 2024 · 2 citations
  3. JUDGEBERT: Assessing Legal Meaning Preservation Between SentencesDavid Beauchemin, Michelle Albert-Rochette, Richard Khoury, Pierre-Luc DézielEMNLP 2025
  4. QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability JudgmentsDavid Beauchemin, Richard KhouryEMNLP 2025

Builds on5

  1. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  2. Revisiting Self-Training for Neural Sequence GenerationJunxian He, Jiatao Gu, Jiajun Shen, Marc'Aurelio RanzatoICLR 2020 · 294 citations
  3. Few-shot Natural Language Generation for Task-Oriented DialogBaolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li et al.EMNLP Findings 2020 · 164 citations
  4. Evaluating the Factual Consistency of Abstractive Text SummarizationWojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard SocherEMNLP 2020 · 67 citations
  5. BLEURT: Learning Robust Metrics for Text GenerationThibault Sellam, Dipanjan Das, Ankur P. ParikhACL 2020 · 40 citations

Related papers

  1. Robots-Dont-Cry: Understanding Falsely Anthropomorphic Utterances in Dialog SystemsDavid Gros, Yu Li, Zhou YuEMNLP 2022 · 8 citations
  2. Towards a Unified Multi-Dimensional Evaluator for Text GenerationMing Zhong, Yang Liu, Da Yin, Yuning Mao et al.EMNLP 2022 · 103 citations
  3. AugNLG: Few-shot Natural Language Generation using Self-trained Data AugmentationXinnuo Xu, Guoyin Wang, Young-Bum Kim, Sungjin LeeACL 2021
  4. ALP: Data Augmentation Using Lexicalized PCFGs for Few-Shot Text ClassificationHazel H. Kim, Daecheol Woo, Seong Joon Oh, Jeong-Won Cha et al.AAAI 2022 · 33 citations
  5. Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation ModelsWangchunshu Zhou, Ke XuAAAI 2020 · 49 citations
Dusk over the sea between two cliffs drawn in fine vertical lines