Lune

ACL Findings2021

TILGAN: Transformer-based Implicit Latent GAN for Diverse and Coherent Text Generation

Shizhe Diao, Xinwei Shen, Kashun Shum, Yan Song, Tong Zhang

Abstract

Conventional autoregressive models have achieved great success in text generation but suffer from the exposure bias problem in that token sequences in the training and in the generation stages are mismatched. While generative adversarial networks (GANs) can remedy this problem, existing implementations of GANs directly on discrete outputs tend to be unstable and lack diversity. In this work, we propose TILGAN, a Transformerbased Implicit Latent GAN, which combines a Transformer autoencoder and GAN in the latent space with a novel design and distribution matching based on the Kullback-Leibler (KL) divergence. Specifically, to improve local and global coherence, we explicitly introduce a multi-scale discriminator to capture the semantic information at varying scales among the sequence of hidden representations encoded by Transformer. Moreover, the decoder is enhanced by an additional KL loss to be consistent with the latent-generator. Experimental results on three benchmark datasets demonstrate the validity and effectiveness of our model, by obtaining significant improvements and a better quality-diversity trade-off in automatic and human evaluation for both unconditional and conditional generation tasks. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention cites Neural Text Generation With Unlikelihood Training
  • TextGAIL: Generative Adversarial Imitation Learning for Text Generation cites Neural Text Generation With Unlikelihood Training
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Modeling Structural Similarities between Documents for Coherence Assessment with Graph Convolutional NetworksWei Liu, Xiyan Fu, Michael StrubeACL 2023 · 3 citations
  2. Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating AttentionWenhao Li, Xiaoyuan Yi, Jinyi Hu, Maosong Sun et al.EMNLP 2022 · 1 citation

Builds on5

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Neural Text Generation With Unlikelihood TrainingSean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan et al.ICLR 2020 · 683 citations
  3. High Fidelity Speech Synthesis with Adversarial NetworksMikolaj Binkowski, Jeff Donahue, Sander Dieleman, Aidan Clark et al.ICLR 2020 · 263 citations
  4. End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2DongHoon Ham, Jeong-Gwan Lee, Youngsoo Jang, Kee-Eung KimACL 2020 · 167 citations
  5. TextGAIL: Generative Adversarial Imitation Learning for Text GenerationQingyang Wu, Lei Li, Zhou YuAAAI 2021 · 54 citations

Related papers

  1. ColdGANs: Taming Language GANs with Cautious Sampling StrategiesThomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski et al.NeurIPS 2020 · 19 citations
  2. Language GANs Falling ShortMassimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle et al.ICLR 2020 · 236 citations
  3. DF-GAN: A Simple and Effective Baseline for Text-to-Image SynthesisMing Tao, Hao Tang, Fei Wu, Xiaoyuan Jing et al.CVPR 2022 · 296 citations
  4. TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale UpYifan Jiang, Shiyu Chang, Zhangyang WangNeurIPS 2021 · 515 citations
  5. ViTGAN: Training GANs with Vision TransformersKwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang et al.ICLR 2022 · 225 citations
Dusk over the sea between two cliffs drawn in fine vertical lines