Lune

EMNLP Findings2024

Empirical Prior for Text Autoencoders

Yongjing Yin, Wenyang Gao, Haodong Wu, Jianhao Yan, Yue Zhang

Abstract

This paper explores the application of Variational Autoencoders (VAE) in text generation, focusing on overcoming challenges like posterior collapse and the limitations of simplistic prior distributions. We investigate a transition from VAE to text autoencoders (AE), which model a compact latent space and preserve the capability of the language model itself. Our method involves layer-wise latent vectors regularized by orthogonal constraints to encourage distinct semantic spaces. In particular, we estimate an empirical prior online from the learned latent vectors to support sampling during generation like VAE. Experimental results on standard benchmarks demonstrate that the autoencoders generate higher quality and more diverse text than the state-of-the-art VAE-based Transformer models, offering an effective alternative for generative language modeling.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers cites Language Models are Few-Shot Learners
  • MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers cites The Curious Case of Neural Text Degeneration
  • Language Models are Few-Shot Learners cites The Curious Case of Neural Text Degeneration
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on14

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  3. NVAE: A Deep Hierarchical Variational AutoencoderArash Vahdat, Jan KautzNeurIPS 2020 · 1,141 citations
  4. MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence FrontiersKrishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun et al.NeurIPS 2021 · 606 citations
  5. From Variational to Deterministic AutoencodersPartha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael J. Black et al.ICLR 2020 · 298 citations

Related papers

  1. Controlled Text Generation Using Dictionary Prior in Variational AutoencodersXianghong Fang, Jian Li, Lifeng Shang, Xin Jiang et al.ACL Findings 2022 · 5 citations
  2. Contrastive Deterministic Autoencoders For Language ModelingAmur Ghose, Pascal PoupartEMNLP Findings 2023 · 1 citation
  3. A Batch Normalized Inference Network Keeps the KL Vanishing AwayQile Zhu, Wei Bi, Xiaojiang Liu, Xiyao Ma et al.ACL 2020 · 70 citations
  4. Improving Variational Autoencoders with Density Gap-based RegularizationJianfei Zhang, Jun Bai, Chenghua Lin, Yanmeng Wang et al.NeurIPS 2022 · 11 citations
  5. Addressing Posterior Collapse with Mutual Information for Improved Variational Neural Machine TranslationArya D. McCarthy, Xian Li, Jiatao Gu, Ning DongACL 2020 · 19 citations
Dusk over the sea between two cliffs drawn in fine vertical lines