On Error Propagation of Diffusion Models

Although diffusion models (DMs) have shown promising performances in a number of tasks (e.g., speech synthesis and image generation), they might suffer from error propagation because of their sequential structure. However, this is not certain because some sequential models, such as Conditional Random Field (CRF), are free from this problem. To address this issue, we develop a theoretical framework to mathematically formulate error propagation in the architecture of DMs, The framework contains three elements, including modular error, cumulative error, and propagation equation. The modular and cumulative errors are related by the equation, which interprets that DMs are indeed affected by error propagation. Our theoretical study also suggests that the cumulative error is closely related to the generation quality of DMs. Based on this finding, we apply the cumulative error as a regularization term to reduce error propagation. Because the term is computationally intractable, we derive its upper bound and design a bootstrap algorithm to efficiently estimate the bound for optimization. We have conducted extensive experiments on multiple image datasets, showing that our proposed regularization reduces error propagation, significantly improves vanilla DMs, and outperforms previous baselines. INTRODUCTION DMs potentially suffer from error propagation. While diffusion models (DMs) have shown impressive performances in a number of tasks (e.g., image synthesis (Rombach et al., 2022) , speech processing (Kong et al., 2021) and natural language generation (Li et al., 2022)), they might still suffer from error propagation (Motter & Lai, 2002) , a classical problem in engineering practices (e.g., communication networks) (Fu et al., 2020; Motter & Lai, 2002; Crucitti et al., 2004) . The problem mainly affects some chain models that consist of many end-to-end connected modules. For those models, the output error from one module will spread to subsequent modules such that errors accumulate along the chain. Since DMs are exactly of a chain structure and have a large number of modules (e.g., 1000 for DDPM (Ho et al., 2020) ), the effect of error propagation on diffusion models is potentially notable and worth a careful study. Current works lack reliable explanations. Some recent works (Ning et al., 2023; Li et al., 2023; Daras et al., 2023) have noticed this problem and named it as exposure bias or sampling drift. However, those works claim that error propagation happens to DMs simply because the models are of a cascade structure. In fact, many sequential models (e.g., CRF (Lafferty et al., 2001) and CTC (Graves et al., 2006)) are free from the problem. Therefore, a solid explanation is expected to answer whether(or even why) error propagation impacts on DMs. Our theory for the error propagation of DMs. One main focus of this work is to develop a theoretical framework for analyzing the error propagation of DMs. With this framework, we can clearly understand how this problem is mathematically formulated in the architecture of DMs and easily see whether it has a significant impact on the models. Our framework contains three elements: modular error, cumulative error, and propagation equation. The first two elements respectively measure the prediction error of one single module and the accumulated error of multiple modules, while the last one tells how these errors are related. We