EXION: Exploiting Inter-and Intra-Iteration Output Sparsity for Diffusion Models

Over the past few years, diffusion models have emerged as novel solutions in AI industries, offering the capability to generate diverse, multi-modal outputs such as images, videos, and motions from input prompts (i.e., text). Despite their impressive capabilities, diffusion models face significant challenges in computing, including excessive latency and energy consumption due to the numerous iterations in model architecture. Although prior works specialized in transformer acceleration can be applied to diffusion models, given that transformers are key components, the problem of the iterative nature of diffusion models still needs to be addressed.In this paper, we present EXION, the first software-hardware co-designed diffusion accelerator that solves the computation challenges of excessive iterations by exploiting the unique inter-and intra-iteration output sparsity in diffusion models. To this end, we propose two software-level optimizations in EXION. First, we introduce the FFN-Reuse algorithm that identifies and skips redundant computations in FFN layers across different iterations (i.e., inter-iteration sparsity). Second, we use a modified eager prediction method that employs two-step leading-one detection to accurately predict the attention score in diffusion models, skipping unnecessary computations within an iteration (i.e., intra-iteration sparsity). We also introduce a novel data compaction mechanism named ConMerge, which can enhance hardware utilization by condensing and merging large and sparse matrices into small and compact forms. Finally, EXION has a dedicated hardware architecture that supports the above sparsity-inducing algorithms, translating high output sparsity into improved energy efficiency and performance. To verify the feasibility of the EXION accelerator, we first demonstrate that it has no impact on accuracy in various types of multi-modal diffusion models, including text-to-motion, -audio, -image, and -video. We then instantiate EXION in both server-and edge-level settings and compare its performance against GPUs with similar specifications. Our evaluation shows that EXION achieves dramatic improvements in performance and energy efficiency by 3.2-379.3 × and 45.1-3067.6 × compared to a server GPU and by 42.6-1090.9 × and 196.9-4668.2 × compared to an edge GPU.