Lune

EMNLP Findings2023

Semi-Structured Object Sequence Encoders

Rudra Murthy V, Riyaz A. Bhat, R. Chulaka Gunasekara, Siva Sankalp Patel, Hui Wan, Tejas I. Dhamecha, Danish Contractor, Marina Danilevsky

Abstract

In this paper we explore the task of modeling semi-structured object sequences; in particular, we focus our attention on the problem of developing a structure-aware input representation for such sequences. Examples of such data include user activity on websites, machine logs, and many others. This type of data is often represented as a sequence of sets of keyvalue pairs over time and can present modeling challenges due to an ever-increasing sequence length thereby affecting the quality of the representation. We propose a two-part approach, which first considers each key independently and encodes a representation of its values over time; we then self-attend over these value-aware key representations to accomplish a downstream task. This allows us to learn better representation while being able to operate on longer object sequences than existing methods. We introduce a novel shared-attentionhead architecture between the two modules and present an innovative training schedule that interleaves the training of both modules with shared weights for some attention heads. 1 Our experiments on multiple prediction tasks using real-world data demonstrate that our approach outperforms a unified network with hierarchical encoding, as well as other methods including a record-centric representation and a flattened representation of the sequence.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting cites Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on15

  1. Informer: Beyond Efficient Transformer for Long Sequence Time-Series ForecastingHaoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang et al.AAAI 2021 · 7,289 citations
  2. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series ForecastingHaixu Wu, Jiehui Xu, Jianmin Wang, Mingsheng LongNeurIPS 2021 · 5,824 citations
  3. AdaShare: Learning What To Share For Efficient Deep Multi-Task LearningXimeng Sun, Rameswar Panda, Rogério Feris, Kate SaenkoNeurIPS 2020 · 337 citations
  4. Sequence-to-Sequence Knowledge Graph Completion and Question AnsweringApoorv Saxena, Adrian Kochsiek, Rainer GemullaACL 2022 · 183 citations
  5. MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual DataYilun Zhao, Yunxiang Li, Chenying Li, Rui ZhangACL 2022 · 168 citations

Related papers

  1. Predicting Temporal Sets with Deep Neural NetworksLe Yu, Leilei Sun, Bowen Du, Chuanren Liu et al.KDD 2020 · 52 citations
  2. Long-Sequence Recommendation Models Need Decoupled EmbeddingsNingya Feng, Junwei Pan, Jialong Wu, Baixu Chen et al.ICLR 2025
  3. AutoAttend: Automated Attention Representation SearchChaoyu Guan, Xin Wang, Wenwu ZhuICML 2021 · 46 citations
  4. Hierarchical Contextualized Representation for Named Entity RecognitionYing Luo, Fengshun Xiao, Hai ZhaoAAAI 2020 · 138 citations
  5. Déjà vu: A Contextualized Temporal Attention Mechanism for Sequential RecommendationJibang Wu, Renqin Cai, Hongning WangWWW 2020 · 66 citations
Dusk over the sea between two cliffs drawn in fine vertical lines