Lune

EMNLP Findings2020

Cascaded Semantic and Positional Self-Attention Network for Document Classification

Juyong Jiang, Jie Zhang, Kai Zhang

Abstract

Transformers have shown great success in learning representations for language modelling. However, an open challenge still remains on how to systematically aggregate semantic information (word embedding) with positional (or temporal) information (word orders). In this work, we propose a new architecture to aggregate the two sources of information using cascaded semantic and positional self-attention network (CSPAN) in the context of document classification. The CSPAN uses a semantic self-attention layer cascaded with Bi-LSTM to process the semantic and positional information in a sequential manner, and then adaptively combine them together through a residual connection. Compared with commonly used positional encoding schemes, CSPAN can exploit the interaction between semantics and word positions in a more interpretable and adaptive manner, and the classification performance can be notably improved while simultaneously preserving a compact model size and high convergence rate. We evaluate the CSPAN model on several benchmark data sets for document classification with careful ablation studies, and demonstrate the encouraging results compared with state of the art.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Related papers

  1. SelfDoc: Self-Supervised Document Representation LearningPeizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I. Morariu et al.CVPR 2021
  2. Recurrent Attention Networks for Long-text ModelingXianming Li, Zongxi Li, Xiaotian Luo, Haoran Xie et al.ACL Findings 2023 · 8 citations
  3. Improving Span Representation by Efficient Span-Level AttentionPengyu Ji, Songlin Yang, Kewei TuEMNLP Findings 2023
  4. Multiple Positional Self-Attention Network for Text ClassificationBiyun Dai, Jinlong Li, Ruoyi XuAAAI 2020 · 10 citations
  5. Positional Encoding for Spiking TransformersZijian Zhou, Yu Liang, Honglin Cao, Ammar Belatreche et al.ICML 2026 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines