Causal Document-Grounded Dialogue Pre-training

The goal of document-grounded dialogue (DocGD) is to generate a response by anchoring the evidence in a supporting document in accordance with the dialogue context. This entails four causally interconnected variables. While task-specific pre-training has significantly enhanced performances on numerous downstream tasks, existing DocGD methods still rely on general pre-trained language models without a specifically tailored pre-training approach that explicitly captures the causal relationships. To address this, we present the first causallycomplete dataset construction strategy for developing million-scale DocGD pre-training corpora. Additionally, we propose a causallyperturbed pre-training strategy to better capture causality by introducing perturbations on the variables and optimizing the overall causal effect. Experiments conducted on three benchmark datasets demonstrate that our causal pretraining yields substantial and consistent improvements in fully-supervised, low-resource, few-shot, and zero-shot settings 1 . * Corresponding author. 1 The datasets and code will be made publicly available. d c r e I heard about a disability benefit for clothing, is it true? Yes, how can I help you? How to apply for the allowance if my dad is eligible? 𝑐 You'll need to file a claim for disability compensation.