Lune

ACL Findings2023

Hypothetical Training for Robust Machine Reading Comprehension of Tabular Context

Moxin Li, Wenjie Wang, Fuli Feng, Hanwang Zhang, Qifan Wang, Tat-Seng Chua

Abstract

Machine Reading Comprehension (MRC) models easily learn spurious correlations from complex contexts such as tabular data.Counterfactual training-using the factual and counterfactual data by augmentation-has become a promising solution.However, it is costly to construct faithful counterfactual examples because it is tricky to maintain the consistency and dependency of the tabular data.In this paper, we take a more efficient fashion to ask hypothetical questions like "in which year would the net profit be larger if the revenue in 2019 were 38,298?",whoseeffectsontheanswersareequivalenttothoseexpensivecounterfactualtables.Weproposeahypotheticaltrainingframeworkthatusespairedexampleswithdifferenthypotheticalquestionstosupervisethedirectionofmodelgradienttowardsthecounterfactualanswerchange.ThesuperiorgeneralizationresultsontabularMRCdatasets,includinganewlyconstructedstresstestandMultiHiertt,validateoureffectiveness.ContextYear20192018Revenue(38,298?", whose effects on the answers are equivalent to those expensive counterfactual tables.We propose a hypothetical training framework that uses paired examples with different hypothetical questions to supervise the direction of model gradient towards the counterfactual answer change.The superior generalization results on tabular MRC datasets, including a newly constructed stress test and MultiHiertt, validate our effectiveness.Context Year 2019 2018 Revenue () 34,298 37,566 Cost ()4,5506,240NetProfit() 4,550 6,240 Net Profit () 29,748 31,326 Question Answer Sample Type In which year was the net profit larger?2018 Factual example In which year would the net profit be larger if the revenue in 2019 were 30,000instead?2018HypotheticalexampleInwhichyearwouldthenetprofitbelargeriftherevenuein2019were30,000 instead?2018 Hypothetical example In which year would the net profit be larger if the revenue in 2019 were 38,298 instead?2019 Hypothetical example Year 2019 2018 Revenue ()38,29837,566Cost() 38,298 37,566 Cost () 4,550 6,240 Net Profit ($) 33,748 31,326 In which year was the net profit larger?2019 Counterfactual example Editing the tabular context is costly due to the dependency across table entries .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Counterfactual Generative Networks cites Learning The Difference That Makes A Difference With Counterfactually-Augmented Data
  • Evaluating Models' Local Decision Boundaries via Contrast Sets cites Learning The Difference That Makes A Difference With Counterfactually-Augmented Data
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on29

  1. Learning The Difference That Makes A Difference With Counterfactually-Augmented DataDivyansh Kaushik, Eduard H. Hovy, Zachary Chase LiptonICLR 2020 · 625 citations
  2. Evaluating Models' Local Decision Boundaries via Contrast SetsMatt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant et al.EMNLP Findings 2020 · 260 citations
  3. HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual DataWenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong et al.EMNLP Findings 2020 · 205 citations
  4. MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual DataYilun Zhao, Yunxiang Li, Chenying Li, Rui ZhangACL 2022 · 168 citations
  5. Counterfactual Generative NetworksAxel Sauer, Andreas GeigerICLR 2021 · 145 citations

Related papers

  1. Realistic Data Augmentation Framework for Enhancing Tabular ReasoningDibyakanti Kumar, Vivek Gupta, Soumya Sharma, Shuo ZhangEMNLP Findings 2022 · 2 citations
  2. Generation of Training Examples for Tabular Natural Language InferenceJean-Flavien Bussotti, Enzo Veltri, Donatello Santoro, Paolo PapottiSIGMOD 2024 · 7 citations
  3. A Robust Adversarial Training Approach to Machine Reading ComprehensionKai Liu, Xin Liu, An Yang, Jing Liu et al.AAAI 2020 · 54 citations
  4. TSQA: Tabular Scenario Based Question AnsweringXiao Li, Yawei Sun, Gong ChengAAAI 2021 · 37 citations
  5. Measuring and Improving Attentiveness to Partial Inputs with CounterfactualsYanai Elazar, Bhargavi Paranjape, Hao Peng, Sarah Wiegreffe et al.EMNLP Findings 2024
Dusk over the sea between two cliffs drawn in fine vertical lines