Look-back Decoding for Open-Ended Text Generation

Given a prefix (context), open-ended generation aims to decode texts that are coherent, which do not abruptly drift from previous topics, and informative, which do not suffer from undesired repetitions. In this paper, we propose Look-back , an improved decoding algorithm that leverages the Kullback-Leibler divergence to track the distribution distance between current and historical decoding steps. Thus Lookback can automatically predict potential repetitive phrase and topic drift, and remove tokens that may cause the failure modes, restricting the next token probability distribution within a plausible distance to the history. We perform decoding experiments on document continuation and story generation, and demonstrate that Look-back is able to generate more fluent and coherent text, outperforming other strong decoding methods significantly in both automatic and human evaluations 1 . Degeneration LM (Decoding) Continuation Prefix: In addition to live broadcasts FIFA Fan Fests offer food and beverages, merchandise and various entertainment events by local and international artists. The start of 2006 World Cup was Repetitive Continuation S1: (GPT2-XL greedy) marked by the FIFA Fan Fest in Sao Paulo, Brazil.The FIFA Fan Fest is a unique event that brings together fans from all over the world to celebrate the FIFA World Cup . The FIFA Fan Fest is a unique event that brings together fans from... S2: (davinci-002 greedy) celebrated with a concert by Bon Jovi in Berlin.The FIFA Fan Fest in Munich was located at the Olympic Park.The FIFA Fan Fest in Frankfurt was located at the Römerberg.The FIFA Fan Fest in Hamburg was located at the ... Prefix: Burkan died from an attack of acute indigestion at his country home in Great Neck on June 6, 1936. Three thousand people attended his funeral at Off-Topic Continuation S3: (ada-001 greedy) the local church.The authorities are investigating how Khedira managed to enter the house and what role he played in the attack.