Lune

ACL Findings2021

How does Attention Affect the Model?

Cheng Zhang, Qiuchi Li, Lingyu Hua, Dawei Song

Abstract

The attention layer has become a prevalent component in improving the effectiveness of neural network models for NLP tasks. Figuring out why attention is effective and its interpretability has attracted a widespread deliberation. Current studies mostly investigate the effect of attention mechanism based on the attention distribution it generates with one single neural network structure. However they do not consider the changes in semantic capability of different components in the model due to the attention mechanism, which can vary across different network structures. In this paper, we propose a comprehensive analytical framework that exploits a convex hull representation of sequence semantics in an n-dimensional Semantic Euclidean Space and defines a series of indicators to capture the impact of attention on sequence semantics. Through a series of experiments on various NLP tasks and three representative recurrent units, we analyze why and how attention benefits the semantic capacity of different types of recurrent neural networks based on the indicators defined in the proposed framework.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Is Attention Explanation? An Introduction to the DebateAdrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens et al.ACL 2022

Related papers

  1. Recurrent Attention Networks for Long-text ModelingXianming Li, Zongxi Li, Xiaotian Luo, Haoran Xie et al.ACL Findings 2023 · 8 citations
  2. ACT: an Attentive Convolutional Transformer for Efficient Text ClassificationPengfei Li, Peixiang Zhong, Kezhi Mao, Dongzhe Wang et al.AAAI 2021 · 47 citations
  3. Quantifying Memory Utilization with Effective State-SizeRom N. Parnichkun, Neehal Tumma, Armin W. Thomas, Alessandro Moro et al.ICML 2025
  4. Exploring Attention Attractors in Large Language ModelsZiheng Wang, Zihao Yue, Wenxuan Wang, Qin JinACL 2026
  5. Recurrent Attention for Neural Machine TranslationJiali Zeng, Shuangzhi Wu, Yongjing Yin, Yufan Jiang et al.EMNLP 2021
Dusk over the sea between two cliffs drawn in fine vertical lines