Lune

EMNLP Findings2024

Topic Modeling: Contextual Token Embeddings Are All You Need

Dimo Angelov, Diana Inkpen

Abstract

The goal of topic modeling is to find meaningful topics that capture the information present in a collection of documents. The main challenges of topic modeling are finding the optimal number of topics, labeling the topics, segmenting documents by topic, and evaluating topic model performance. Current neural approaches have tackled some of these problems but none have been able to solve all of them. We introduce a novel topic modeling approach, Contextual-Top2Vec, which uses document contextual token embeddings, it creates hierarchical topics, finds topic spans within documents and labels topics with phrases rather than just words. We propose the use of BERTScore to evaluate topic coherence and to evaluate how informative topics are of the underlying documents. Our model outperforms the current state-of-the-art models on a comprehensive set of topic model evaluation metrics.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Large Language Models Struggle to Describe the Haystack without Human Help: A Social Science-Inspired Evaluation of Topic Models cites BERTScore: Evaluating Text Generation with BERT
  • Large Language Models Struggle to Describe the Haystack without Human Help: A Social Science-Inspired Evaluation of Topic Models cites Is Automated Topic Model Evaluation Broken? The Incoherence of Coherence
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Neural Topic Modeling with Large Language Models in the LoopXiaohao Yang, He Zhao, Weijie Xu, Yuanyuan Qi et al.ACL 2025 · 13 citations
  2. MGT-Prism: Enhancing Domain Generalization for Machine-Generated Text Detection via Spectral AlignmentShengchao Liu, Xiaoming Liu, Chengzhengxu Li, Zhaohan Zhang et al.AAAI 2026 · 1 citation
  3. Large Language Models Struggle to Describe the Haystack without Human Help: A Social Science-Inspired Evaluation of Topic ModelsZongxia Li, Lorena Calvo-Bartolomé, Alexander Miserlis Hoyle, Paiheng Xu et al.ACL 2025
  4. Bridging the Gap: Cardinality Estimation for Semantic Queries on Unstructured DataShihui Xu, Jiayi Wang, Guoliang LiSIGMOD 2026

Builds on6

  1. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  2. Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
  3. MPNet: Masked and Permuted Pre-training for Language UnderstandingKaitao Song, Xu Tan, Tao Qin, Jianfeng Lu et al.NeurIPS 2020 · 1,957 citations
  4. Is Automated Topic Model Evaluation Broken? The Incoherence of CoherenceAlexander Miserlis Hoyle, Pranav Goel, Andrew Hian-Cheong, Denis Peskov et al.NeurIPS 2021 · 220 citations
  5. Multi-View Document Representation Learning for Open-Domain Dense RetrievalShunyu Zhang, Yaobo Liang, Ming Gong, Daxin Jiang et al.ACL 2022 · 80 citations

Related papers

  1. Benchmarking Neural Topic Models: An Empirical StudyThanh-Nam Doan, Tuan-Anh HoangACL Findings 2021 · 27 citations
  2. HiCOT: Improving Neural Topic Models via Optimal Transport and Contrastive LearningHoang Tran Vuong, Tue Le, Tu Vu, Tung Nguyen et al.ACL Findings 2025 · 3 citations
  3. Representing Mixtures of Word Embeddings with Mixtures of Topic EmbeddingsDongsheng Wang, Dandan Guo, He Zhao, Huangjie Zheng et al.ICLR 2022 · 56 citations
  4. Neural Attention-Aware Hierarchical Topic ModelYuan Jin, He Zhao, Ming Liu, Lan Du et al.EMNLP 2021
  5. S³ - Semantic Signal SeparationMárton Kardos, Jan Kostkan, Kenneth C. Enevoldsen, Arnault-Quentin Vermillet et al.ACL 2025
Dusk over the sea between two cliffs drawn in fine vertical lines