Lune

ACL Findings2023

Dynamic Structured Neural Topic Model with Self-Attention Mechanism

Nozomu Miyamoto, Masaru Isonuma, Sho Takase, Junichiro Mori, Ichiro Sakata

Abstract

This study presents a dynamic structured neural topic model, which can handle the timeseries development of topics while capturing their dependencies. Our model captures the topic branching and merging processes by modeling topic dependencies based on a selfattention mechanism. Additionally, we introduce citation regularization, which induces attention weights to represent citation relations by modeling text and citations jointly. Our model outperforms a prior dynamic embedded topic model (Dieng et al., 2019) regarding perplexity and coherence, while maintaining sufficient diversity across topics. Furthermore, we confirm that our model can potentially predict emerging topics from academic literature.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Neural Topic Modeling with Large Language Models in the LoopXiaohao Yang, He Zhao, Weijie Xu, Yuanyuan Qi et al.ACL 2025 · 13 citations
  2. Dynamics of Spontaneous Topic Changes in Next Token Prediction with Self-AttentionMumin Jia, Jairo Diaz RodriguezNeurIPS 2025 · 3 citations
  3. DVI-DTM: Dual-View Representation Learning for Interpretable Short Text Dynamic Topic ModelingDi Liu, Zheng Fang, Bin WuACL 2026

Builds on3

  1. S2ORC: The Semantic Scholar Open Research CorpusKyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney et al.ACL 2020 · 424 citations
  2. Dynamic Topic Models for Temporal Document NetworksDelvin Ce Zhang, Hady W. LauwICML 2022 · 26 citations
  3. Tree-Structured Topic Modeling with Nonparametric Neural Variational InferenceZiye Chen, Cheng Ding, Zusheng Zhang, Yanghui Rao et al.ACL 2021

Related papers

  1. Dynamic Multi-Context Attention Networks for Citation Forecasting of Scientific PublicationsTaoran Ji, Nathan Self, Kaiqun Fu, Zhiqian Chen et al.AAAI 2021 · 6 citations
  2. Effective Neural Topic Modeling with Embedding Clustering RegularizationXiaobao Wu, Xinshuai Dong, Thong Thanh Nguyen, Anh Tuan LuuICML 2023 · 87 citations
  3. Neural Dynamic Focused Topic ModelKostadin Cvejoski, Ramsés J. Sánchez, César OjedaAAAI 2023 · 9 citations
  4. Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word ExclusionXiaobao Wu, Xinshuai Dong, Liangming Pan, Thong Nguyen et al.ACL Findings 2024 · 9 citations
  5. An Attentional Multi-scale Co-evolving Model for Dynamic Link PredictionGuozhen Zhang, Tian Ye, Depeng Jin, Yong LiWWW 2023 · 26 citations
Dusk over the sea between two cliffs drawn in fine vertical lines