Language modeling via stochastic processes

Modern language models can generate high-quality short texts. However, they often meander or are incoherent when generating longer texts. These issues arise from the next-token-only language modeling objective. Recent work in selfsupervised learning suggests that models can learn good latent representations via contrastive learning, which can be effective for discriminative tasks. Our work analyzes the application of contrastive representations for generative tasks, like long text generation. We propose one approach for leveraging constrastive representations, which we call Time Control (TC). TC first learns a contrastive representation of the target text domain, then generates text by decoding from these representations. Compared to domain-specific methods and fine-tuning GPT2 across a variety of text domains, TC performs competitively to methods specific for learning sentence representations on discourse coherence. On long text generation settings, TC preserves the text structure both in terms of ordering (up to +15% better) and text length consistency (up to +90% better) 1 . 1 Please find our code at https://github.com/rosewang2008/language_modeling_via_ stochastic_processes Correction note This is a revised version of the original ICLR 2022 paper. During post-publication code review, we discovered that the original version of the code did not leverage goal-directedness during decoding. While we still find that contrastive representations lead to gains in our evaluations, this error affects other claims made in the paper on goal-directed decoding. To correct this and help improve our understanding of goal-directed decoding, this updated version of the manuscript contains results on both goal-directed and non goal-directed baselines. We detail the difference between the original work and this updated work in Appendix H.