ACL Findings2021
How does Attention Affect the Model?
Abstract
The attention layer has become a prevalent component in improving the effectiveness of neural network models for NLP tasks. Figuring out why attention is effective and its interpretability has attracted a widespread deliberation. Current studies mostly investigate the effect of attention mechanism based on the attention distribution it generates with one single neural network structure. However they do not consider the changes in semantic capability of different components in the model due to the attention mechanism, which can vary across different network structures. In this paper, we propose a comprehensive analytical framework that exploits a convex hull representation of sequence semantics in an n-dimensional Semantic Euclidean Space and defines a series of indicators to capture the impact of attention on sequence semantics. Through a series of experiments on various NLP tasks and three representative recurrent units, we analyze why and how attention benefits the semantic capacity of different types of recurrent neural networks based on the indicators defined in the proposed framework.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Related papers
- Recurrent Attention Networks for Long-text ModelingXianming Li, Zongxi Li, Xiaotian Luo, Haoran Xie et al.ACL Findings 2023 · 8 citations
- ACT: an Attentive Convolutional Transformer for Efficient Text ClassificationPengfei Li, Peixiang Zhong, Kezhi Mao, Dongzhe Wang et al.AAAI 2021 · 47 citations
- Quantifying Memory Utilization with Effective State-SizeRom N. Parnichkun, Neehal Tumma, Armin W. Thomas, Alessandro Moro et al.ICML 2025
- Exploring Attention Attractors in Large Language ModelsZiheng Wang, Zihao Yue, Wenxuan Wang, Qin JinACL 2026
- Recurrent Attention for Neural Machine TranslationJiali Zeng, Shuangzhi Wu, Yongjing Yin, Yufan Jiang et al.EMNLP 2021
