Lune

EMNLP Findings2020

Why and when should you pool? Analyzing Pooling in Recurrent Architectures

Pratyush Maini, Keshav Kolluru, Danish Pruthi, Mausam

Abstract

Pooling-based recurrent neural architectures consistently outperform their counterparts without pooling on sequence classification tasks. However, the reasons for their enhanced performance are largely unexamined. In this work, we explore three commonly used pooling techniques (mean-pooling, max-pooling, and attention 1 ), and propose max-attention, a novel variant that captures interactions among predictive tokens in a sentence. Using novel experiments, we demonstrate that pooling architectures substantially differ from their nonpooling equivalents in their learning ability and positional biases: (i) pooling facilitates better gradient flow than BiLSTMs in initial training epochs, and (ii) BiLSTMs are biased towards tokens at the beginning and end of the input, whereas pooling alleviates this bias. Consequently, we find that pooling yields large gains in low resource scenarios, and instances when salient words lie towards the middle of the input. Across several text classification tasks, we find max-attention to frequently outperform other pooling techniques. 2

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. "Covid vaccine is against Covid but Oxford vaccine is made at Oxford!" Semantic Interpretation of Proper Noun CompoundsKeshav Kolluru, Gabriel Stanovsky, MausamEMNLP 2022 · 1 citation

Builds on1

  1. Learning to Deceive with Attention-Based ExplanationsDanish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig et al.ACL 2020 · 17 citations

Related papers

  1. Attentive Pooling with Learnable Norms for Text RepresentationChuhan Wu, Fangzhao Wu, Tao Qi, Xiaohui Cui et al.ACL 2020 · 17 citations
  2. Less Mature is More Adaptable for Sentence-level Language ModelingAbhilasha Sancheti, David Dale, Artyom Kozhevnikov, Maha ElbayadACL 2025
  3. Efficient Representation Learning via Adaptive Context PoolingChen Huang, Walter Talbott, Navdeep Jaitly, Joshua M. SusskindICML 2022 · 10 citations
  4. Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM EmbeddingsXueying Ding, Xingyue Huang, Mingxuan Ju, Liam Collins et al.ACL 2026 · 3 citations
  5. A Joint Model for Document Segmentation and Segment LabelingJoe Barrow, Rajiv Jain, Vlad I. Morariu, Varun Manjunatha et al.ACL 2020 · 47 citations
Dusk over the sea between two cliffs drawn in fine vertical lines