Relevance-Promoting Language Model for Short-Text Conversation
Abstract
Despite the effectiveness of sequence-to-sequence framework on the task of Short-Text Conversation (STC), the issue of under-exploitation of training data (i.e., the supervision signals from query text is ignored) still remains unresolved. Also, the adopted maximization-based decoding strategies, inclined to generating the generic responses or responses with repetition, are unsuited to the STC task. In this paper, we propose to formulate the STC task as a language modeling problem and tailor-make a training strategy to adapt a language model for response generation. To enhance generation performance, we design a relevance-promoting transformer language model, which performs additional supervised source attention after the self-attention to increase the importance of informative query tokens in calculating the token-level representation. The model further refines the query representation with relevance clues inferred from its multiple references during training. In testing, we adopt a randomization-over-maximization strategy to reduce the generation of generic responses. Experimental results on a large Chinese STC dataset demonstrate the superiority of the proposed model on relevance metrics and diversity metrics.1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on1
Related papers
- P3LM: Probabilistically Permuted Prophet Language Modeling for Generative Pre-TrainingJunwei Bao, Yifan Wang, Ying Jiangyong, Yeyun Gong et al.EMNLP Findings 2022
- SMRTer Chatbots: Improving Non-Task-Oriented Dialog with Simulated Multi-Reference TrainingHuda Khayrallah, João SedocEMNLP Findings 2020 · 3 citations
- Chinese Spelling Correction as Rephrasing Language ModelLinfeng Liu, Hongqiu Wu, Hai ZhaoAAAI 2024 · 36 citations
- GRET: Global Representation Enhanced TransformerRongxiang Weng, Hao-Ran Wei, Shujian Huang, Heng Yu et al.AAAI 2020 · 9 citations
- Semi-Supervised Synthetic Data Generation with Fine-Grained Relevance Control for Short Video Search Relevance ModelingHaoran Li, Zhiming Su, Junyan Yao, Enwei Zhang et al.AAAI 2026
