Dynamic Structured Neural Topic Model with Self-Attention Mechanism

This study presents a dynamic structured neural topic model, which can handle the timeseries development of topics while capturing their dependencies. Our model captures the topic branching and merging processes by modeling topic dependencies based on a selfattention mechanism. Additionally, we introduce citation regularization, which induces attention weights to represent citation relations by modeling text and citations jointly. Our model outperforms a prior dynamic embedded topic model (Dieng et al., 2019) regarding perplexity and coherence, while maintaining sufficient diversity across topics. Furthermore, we confirm that our model can potentially predict emerging topics from academic literature.