ACL Findings2025
Explicit Bayesian Inference to Uncover the Latent Themes of Large Language Models
Abstract
Large language models (LLMs) have demonstrated impressive generative capabilities, yet their inner mechanisms remain largely opaque. In this work, we introduce a novel approach to interpret LLMs generation process through the lens of an explicit Bayesian framework by inferring latent topic variables via variational inference. Specifically, we leverage a variational autoencoder-based neural topic model to dynamically approximate the posterior distribution over the high-level latent topic variables at each generation step. By reconstructing the LLM's next-token predictions through these latent topics and maintaining a regularized latent space, our method yields interpretable and diverse topic representations but also has the ability to effectively captures semantic shifts throughout the text. We validate our approach on multiple datasets, showing that our latent topics outperform state-of-the-art topic models on intrinsic measures of coherence and diversity. Furthermore, we demonstrate the utility of our approach in downstream applications by using the inferred topic distributions to retrieve relevant demonstration examples for in-context learning, resulting in significant gains on classification and summarization tasks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Improving Topic Modeling by Distilling Soft Labels from Language Models cites An Explanation of In-context Learning as Implicit Bayesian Inference
- Improving Topic Modeling by Distilling Soft Labels from Language Models cites Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning
- Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning cites An Explanation of In-context Learning as Implicit Bayesian Inference
- Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning cites Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Cited by top-tier papers2
- ANCHOR: Abductive Network Construction with Hierarchical Orchestration for Reliable Probability Inference in Large Language ModelsWentao Qiu, Guanran Luo, Zhongquan Jian, Jingqi Gao et al.ICML 2026 · 2 citations
- Improving Topic Modeling by Distilling Soft Labels from Language ModelsRaymond Li, Amirhossein Abaskohi, Chuyuan Li, Gabriel Murray et al.ICML 2026
Builds on9
- Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order SensitivityYao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel et al.ACL 2022 · 1,494 citations
- An Explanation of In-context Learning as Implicit Bayesian InferenceSang Michael Xie, Aditi Raghunathan, Percy Liang, Tengyu MaICLR 2022 · 1,030 citations
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe et al.EMNLP 2022 · 634 citations
- Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context LearningXinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers et al.NeurIPS 2023 · 206 citations
- DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLMWeijie Xu, Wenxiang Hu, Fanyou Wu, Srinivasan H. SengameduEMNLP Findings 2023 · 20 citations
Related papers
- Latent Concept Disentanglement in Transformer-based Language ModelsGuanzhe Hong, Bhavya Vasudeva, Vatsal Sharan, Cyrus Rashtchian et al.ICLR 2026 · 4 citations
- A Discrete Variational Recurrent Topic Model without the Reparametrization TrickMehdi Rezaee, Francis FerraroNeurIPS 2020 · 31 citations
- On the In-context Generation of Language ModelsZhongtao Jiang, Yuanzhe Zhang, Kun Luo, Xiaowei Yuan et al.EMNLP 2024
- Topic-VQ-VAE: Leveraging Latent Codebooks for Flexible Topic-Guided Document GenerationYoungjoon Yoo, Jongwon ChoiAAAI 2024 · 8 citations
- Revisiting Demonstration Selection Strategies in In-Context LearningKeqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu et al.ACL 2024
