EMNLP Findings2021
Using Question Answering Rewards to Improve Abstractive Summarization
R. Chulaka Gunasekara, Guy Feigenblat, Benjamin Sznajder, Ranit Aharonov, Sachindra Joshi
Abstract
Neural abstractive summarization models have drastically improved in the recent years. However, the summaries generated by these models generally suffer from issues such as: not capturing the critical facts in source documents, and containing facts that are inconsistent with the source documents. In this work, we present a general framework to train abstractive summarization models to alleviate such issues. We first train a sequence-to-sequence model to summarize documents, and then further train this model in a Reinforcement Learning setting with question-answering based rewards. We evaluate the summaries generated by the this framework using multiple automatic measures and human judgements. The experimental results show that the question-answering rewards can be used as a general framework to improve neural abstractive summarization. Particularly, the results from human evaluations show that the summaries generated by our approach are preferred over 30% of the time over the summaries generated by general abstractive summarization models.
Original Document/Dialog Charlee: I'm in class. Theatre in Portuguese lol. Curtis: Realllly? Charlee: Yes. One of my subjects at the university that I attend is portuguese theatre. Charlee: We are preparing for a performance. Curtis: What performance is this? Are you devising it? Charlee: A polish one translated into portuguese. Curtis: Thats quite cool. Who is the writer? Charlee: Mrożek.
Charlee is attending Portuguese theater as a subject at university. He and other students are preparing a play by Mrożek translated into Portuguese.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
7 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback cites Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
- Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
- Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback cites FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
- X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization cites Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
- X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
- Asking and Answering Questions to Evaluate the Factual Consistency of Summaries cites The Curious Case of Neural Text Degeneration
- Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards cites Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization cites The Curious Case of Neural Text Degeneration
Cited by top-tier papers7
- Factually Consistent Summarization via Reinforcement Learning with Textual Entailment FeedbackPaul Roit, Johan Ferret, Lior Shani, Roee Aharoni et al.ACL 2023 · 21 citations
- X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive SummarizationSubhajit Chaudhury, Sarathkrishna Swaminathan, R. Chulaka Gunasekara, Maxwell Crouse et al.EMNLP 2022 · 13 citations
- RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language ModelsYufei Li, Zexin Li, Wei Yang, Cong LiuRTSS 2023 · 10 citations
- Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple RewardsHeejin Do, Sangwon Ryu, Gary Geunbae LeeEMNLP 2024 · 5 citations
- Reward Gaming in Conditional Text GenerationRichard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur P. Parikh et al.ACL 2023 · 2 citations
Builds on9
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive SummarizationJingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. LiuICML 2020 · 2,453 citations
- Asking and Answering Questions to Evaluate the Factual Consistency of SummariesAlex Wang, Kyunghyun Cho, Mike LewisACL 2020 · 317 citations
- Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze RewardLuyang Huang, Lingfei Wu, Lu WangACL 2020 · 152 citations
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive SummarizationEsin Durmus, He He, Mona T. DiabACL 2020 · 90 citations
Related papers
- Keyword-aware Abstractive Summarization by Extracting Set-level Intermediate SummariesYizhu Liu, Qi Jia, Kenny Q. ZhuWWW 2021 · 14 citations
- Guiding Abstractive Dialogue Summarization with Content PlanningYe Wang, Xiaojun Wan, Zhiping CaiEMNLP Findings 2022 · 1 citation
- Generating Multiple-Length Summaries via Reinforcement Learning for Unsupervised Sentence SummarizationDongmin Hyun, Xiting Wang, Chanyoung Park, Xing Xie et al.EMNLP Findings 2022 · 11 citations
- Multi-Fact Correction in Abstractive Text SummarizationYue Dong, Shuohang Wang, Zhe Gan, Yu Cheng et al.EMNLP 2020 · 99 citations
- Learning with Rejection for Abstractive Text SummarizationMeng Cao, Yue Dong, Jingyi He, Jackie Chi Kit CheungEMNLP 2022 · 10 citations
