Lune

AAAI2023Top-tier venue

Less Is More Important: An Attention Module Guided by Probability Density Function for Convolutional Neural Networks

Jingfen Xie, Jian Zhang

Abstract

Attention modules, which adaptively weight and refine features according to the importance of the input, have become a critical technique to boost the capability of convolutional neural networks. However, most existing attention modules are heuristic without a sound interpretation, and thus, require empirical engineering to design structure and operators within the modules. To handle the above issue, based on our 'less is more important' observation, we propose an Attention Module guided by Probability Density Function (PDF), dubbed PdfAM, which enjoys a rational motivation and requires few empirical structure designs. Concretely, we observe that pixels with less occurrence are prone to be textural details or foreground objects with much importance to aid vision tasks. Thus, with PDF values adopted as a smooth and anti-noise alternative to the pixel occurrence frequency, we design our PdfAM by first estimating the PDF based on some distribution assumption, and then predicting a 3D attention map via applying a negative correlation between the attention weights and the estimated PDF values. Furthermore, we develop learnable PDF-rescale parameters so as to adaptively transform the estimated PDF and predict a customized negative correlation. Experiments show that our PdfAM consistently boosts various networks under both high-and low-level vision tasks, and also performs favorably against other attention modules in terms of accuracy and convergence.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • EAN: An Efficient Attention Module Guided by Normalization for Deep Neural Networks cites SRM: A Style-Based Recalibration Module for Convolutional Neural Networks
  • SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural Networks cites SRM: A Style-Based Recalibration Module for Convolutional Neural Networks
  • SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural Networks cites Searching for MobileNetV3
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. EAN: An Efficient Attention Module Guided by Normalization for Deep Neural NetworksJiafeng Li, Zelin Li, Ying WenAAAI 2024 · 2 citations

Builds on8

  1. Searching for MobileNetV3Andrew Howard, Ruoming Pang, Hartwig Adam, Quoc V. Le et al.ICCV 2019 · 9,163 citations
  2. SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural NetworksLingxiao Yang, Ru-Yuan Zhang, Lida Li, Xiaohua XieICML 2021 · 1,593 citations
  3. SRM: A Style-Based Recalibration Module for Convolutional Neural NetworksHyunJae Lee, Hyo-Eun Kim, Hyeonseob NamICCV 2019 · 286 citations
  4. Memory-Augmented Deep Unfolding Network for Compressive SensingJiechong Song, Bin Chen, Jian ZhangACM MM 2021 · 117 citations
  5. Mix-order Attention Networks for Image RestorationTao Dai, Yalei Lv, Bin Chen, Zhi Wang et al.ACM MM 2021 · 4 citations

Related papers

  1. Pseudo 3D Auto-Correlation Network for Real Image DenoisingXiaowan Hu, Ruijun Ma, Zhihong Liu, Yuanhao Cai et al.CVPR 2021
  2. PDD-GAN: Prior-based GAN Network with Decoupling Ability for Single Image DehazingXiaoxuan Chai, Junchi Zhou, Hang Zhou, Jui-Hsin LaiACM MM 2022 · 7 citations
  3. PartialNet: Compute Less, Perform BetterHaiduo Huang, Tian Xia, Wenzhe Zhao, Pengju RenAAAI 2026
  4. Attention Scaling for Crowd CountingXiaoheng Jiang, Li Zhang, Mingliang Xu, Tianzhu Zhang et al.CVPR 2020
  5. Learning Scales from Points: A Scale-aware Probabilistic Model for Crowd CountingZhiheng Ma, Xing Wei, Xiaopeng Hong, Yihong GongACM MM 2020 · 39 citations
Dusk over the sea between two cliffs drawn in fine vertical lines