Lune

ACL Findings2021

Discovering Topics in Long-tailed Corpora with Causal Intervention

Xiaobao Wu, Chunping Li, Yishu Miao

Abstract

Topic models are effective in capturing the latent semantics of large-scale textual data while existing methods are normally designed and evaluated on balanced corpora. However, it contradicts the fact that general corpora in our world are naturally long-tailed, and the longtailed bias can highly impair the topic modeling performance. Therefore, in this paper, we propose a causal inference framework to explain and overcome the issues of topic modeling on long-tailed corpora. In a neat and elegant way, causal intervention is applied in training to take out the influence brought by the long-tailed bias. Extensive experiments on manually constructed and naturally collected datasets demonstrate that our model can mitigate the bias effect, greatly improve topic quality and better discover the hidden semantics on the tail.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

9 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modeling cites Short Text Topic Modeling with Topic Distribution Quantization and Negative Sampling Decoder
  • InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modeling cites Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning
  • On the Affinity, Rationality, and Diversity of Hierarchical Topic Modeling cites InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modeling
  • On the Affinity, Rationality, and Diversity of Hierarchical Topic Modeling cites Short Text Topic Modeling with Topic Distribution Quantization and Negative Sampling Decoder
  • On the Affinity, Rationality, and Diversity of Hierarchical Topic Modeling cites Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning
  • On the Affinity, Rationality, and Diversity of Hierarchical Topic Modeling cites Effective Neural Topic Modeling with Embedding Clustering Regularization
  • Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning cites Short Text Topic Modeling with Topic Distribution Quantization and Negative Sampling Decoder
  • FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model cites InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modeling
  • FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model cites Short Text Topic Modeling with Topic Distribution Quantization and Negative Sampling Decoder
  • FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model cites On the Affinity, Rationality, and Diversity of Hierarchical Topic Modeling
  • FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model cites Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning
  • FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model cites Effective Neural Topic Modeling with Embedding Clustering Regularization
  • Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect cites Decoupling Representation and Classifier for Long-Tailed Recognition
  • Effective Neural Topic Modeling with Embedding Clustering Regularization cites InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modeling
  • Effective Neural Topic Modeling with Embedding Clustering Regularization cites Short Text Topic Modeling with Topic Distribution Quantization and Negative Sampling Decoder
  • Effective Neural Topic Modeling with Embedding Clustering Regularization cites Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers9

  1. Effective Neural Topic Modeling with Embedding Clustering RegularizationXiaobao Wu, Xinshuai Dong, Thong Thanh Nguyen, Anh Tuan LuuICML 2023 · 87 citations
  2. FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic ModelXiaobao Wu, Thong Nguyen, Delvin Zhang, William Yang Wang et al.NeurIPS 2024 · 67 citations
  3. Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive LearningXiaobao Wu, Anh Tuan Luu, Xinshuai DongEMNLP 2022 · 37 citations
  4. InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic ModelingXiaobao Wu, Xinshuai Dong, Thong Nguyen, Chaoqun Liu et al.AAAI 2023 · 35 citations
  5. On the Affinity, Rationality, and Diversity of Hierarchical Topic ModelingXiaobao Wu, Fengjun Pan, Thong Nguyen, Yichao Feng et al.AAAI 2024 · 35 citations

Builds on6

  1. Decoupling Representation and Classifier for Long-Tailed RecognitionBingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan et al.ICLR 2020 · 1,496 citations
  2. Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal EffectKaihua Tang, Jianqiang Huang, Hanwang ZhangNeurIPS 2020 · 533 citations
  3. De-Biased Court's View Generation with CausalityYiquan Wu, Kun Kuang, Yating Zhang, Xiaozhong Liu et al.EMNLP 2020 · 71 citations
  4. Short Text Topic Modeling with Topic Distribution Quantization and Negative Sampling DecoderXiaobao Wu, Chunping Li, Yan Zhu, Yishu MiaoEMNLP 2020 · 61 citations
  5. Counterfactual Generator: A Weakly-Supervised Method for Named Entity RecognitionXiangji Zeng, Yunliang Li, Yuchen Zhai, Yin ZhangEMNLP 2020 · 55 citations

Related papers

  1. Rethinking the Bias of Foundation Model under Long-tailed DistributionJiahao Chen, Bin Qin, Jiangmeng Li, Hao Chen et al.ICML 2025
  2. Delving into Semantic Scale ImbalanceYanbiao Ma, Licheng Jiao, Fang Liu, Yuxin Li et al.ICLR 2023 · 5 citations
  3. LTGC: Long-Tail Recognition via Leveraging LLMs-Driven Generated ContentQihao Zhao, Yalun Dai, Hao Li, Wei Hu et al.CVPR 2024 · 22 citations
  4. Long-tailed Recognition with Model RebalancingJiaan Luo, Feng Hong, Qiang Hu, Xiaofeng Cao et al.NeurIPS 2025 · 12 citations
  5. Adaptive Token Refinement in Long-Tailed Large Vision-Language Models Fine-TuningWenjun Miao, Mingda Li, Yanchao Hao, Zheng WeiICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines