EMNLP Findings2024
Empirical Prior for Text Autoencoders
Yongjing Yin, Wenyang Gao, Haodong Wu, Jianhao Yan, Yue Zhang
Abstract
This paper explores the application of Variational Autoencoders (VAE) in text generation, focusing on overcoming challenges like posterior collapse and the limitations of simplistic prior distributions. We investigate a transition from VAE to text autoencoders (AE), which model a compact latent space and preserve the capability of the language model itself. Our method involves layer-wise latent vectors regularized by orthogonal constraints to encourage distinct semantic spaces. In particular, we estimate an empirical prior online from the learned latent vectors to support sampling during generation like VAE. Experimental results on standard benchmarks demonstrate that the autoencoders generate higher quality and more diverse text than the state-of-the-art VAE-based Transformer models, offering an effective alternative for generative language modeling.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers cites Language Models are Few-Shot Learners
- MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers cites The Curious Case of Neural Text Degeneration
- Language Models are Few-Shot Learners cites The Curious Case of Neural Text Degeneration
Builds on14
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- NVAE: A Deep Hierarchical Variational AutoencoderArash Vahdat, Jan KautzNeurIPS 2020 · 1,141 citations
- MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence FrontiersKrishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun et al.NeurIPS 2021 · 606 citations
- From Variational to Deterministic AutoencodersPartha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael J. Black et al.ICLR 2020 · 298 citations
Related papers
- Controlled Text Generation Using Dictionary Prior in Variational AutoencodersXianghong Fang, Jian Li, Lifeng Shang, Xin Jiang et al.ACL Findings 2022 · 5 citations
- Contrastive Deterministic Autoencoders For Language ModelingAmur Ghose, Pascal PoupartEMNLP Findings 2023 · 1 citation
- A Batch Normalized Inference Network Keeps the KL Vanishing AwayQile Zhu, Wei Bi, Xiaojiang Liu, Xiyao Ma et al.ACL 2020 · 70 citations
- Improving Variational Autoencoders with Density Gap-based RegularizationJianfei Zhang, Jun Bai, Chenghua Lin, Yanmeng Wang et al.NeurIPS 2022 · 11 citations
- Addressing Posterior Collapse with Mutual Information for Improved Variational Neural Machine TranslationArya D. McCarthy, Xian Li, Jiatao Gu, Ning DongACL 2020 · 19 citations
