Lune

ACL Findings2023

TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model

Patrick Kahardipraja, Brielen Madureira, David Schlangen

Abstract

Language is by its very nature incremental in how it is produced and processed. This property can be exploited by NLP systems to produce fast responses, which has been shown to be beneficial for real-time interactive applications. Recent neural network-based approaches for incremental processing mainly use RNNs or Transformers. RNNs are fast but monotonic (cannot correct earlier output, which can be necessary in incremental processing). Transformers, on the other hand, consume whole sequences, and hence are by nature non-incremental. A restartincremental interface that repeatedly passes longer input prefixes can be used to obtain partial outputs, while providing the ability to revise. However, this method becomes costly as the sentence grows longer. In this work, we propose the Two-pass model for AdaPtIve Revision (TAPIR) and introduce a method to obtain an incremental supervision signal for learning an adaptive revision policy. Experimental results on sequence labelling show that our model has better incremental performance and faster inference speed compared to restartincremental Transformers, while showing little degradation on full sequences. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Turning Fixed to Adaptive: Integrating Post-Evaluation into Simultaneous Machine Translation cites Monotonic Multihead Attention
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on6

  1. Transformers are RNNs: Fast Autoregressive Transformers with Linear AttentionAngelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François FleuretICML 2020 · 2,665 citations
  2. Monotonic Multihead AttentionXutai Ma, Juan Miguel Pino, James Cross, Liezl Puzon et al.ICLR 2020 · 148 citations
  3. Approximating Stacked and Bidirectional Recurrent Architectures with the Delayed Recurrent Neural NetworkJavier Turek, Shailee Jain, Vy A. Vo, Mihai Capota et al.ICML 2020 · 13 citations
  4. Turning Fixed to Adaptive: Integrating Post-Evaluation into Simultaneous Machine TranslationShoutao Guo, Shaolei Zhang, Yang FengEMNLP Findings 2022 · 10 citations
  5. Best of Both Worlds: Making High Accuracy Non-incremental Transformer-based Disfluency Detection IncrementalMorteza Rohanian, Julian HoughACL 2021

Related papers

  1. Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLUBrielen Madureira, David SchlangenEMNLP 2020
  2. When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-IncrementalityBrielen Madureira, Patrick Kahardipraja, David SchlangenACL 2024
  3. NADIR: Differential Attention Flow for Non-Autoregressive Transliteration in Indic LanguagesLakshya Tomar, Vinayak Abrol, Puneet AgarwalAAAI 2026
  4. Memory Efficient Continual Learning with TransformersBeyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal et al.NeurIPS 2022 · 75 citations
  5. Learn or Recall? Revisiting Incremental Learning with Pre-trained Language ModelsJunhao Zheng, Shengjie Qiu, Qianli MaACL 2024
Dusk over the sea between two cliffs drawn in fine vertical lines