SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation

Fair comparison of AR and MDLM training efficiency. We conduct controlled experiments comparing autoregressive (AR) models with masked diffusion language models (MDLMs), using the same training data and recommended hyperparameters. This provides the first realworld, meaningful evidence that AR models deliver substantially higher training efficiency than MDLMs under identical computational settings. 2. Adaptation of SDAR from AR and MDLM bases. We systematically evaluate the feasibility of adapting SDAR from both AR and MDLM backbones. Using a 100B-token subset of the 1T training corpus, we perform thorough downstream task benchmarks, demonstrating that AR-based adaptation consistently yields superior performance and is therefore the preferred choice for serving as the base model. Scaling experiments and adaptation recipe. We extend our study to larger models by adapting state-of-the-art open-source AR models (both dense and MoE variants) using an arbitrary 50Btoken open-source dataset. We provide a practical recipe showing that any AR decoder-only model, regardless of its architecture, can be efficiently adapted into SDAR-even without access to its original pretraining data. We argue that this adaptation requires significantly less data than adaptation method such as Dream, making it broadly accessible to the community. 4. Comprehensive analysis of scaling laws. We offer detailed empirical analysis of the relationship between model size, block size, downstream task performence and inference speed. Our study spans models from 1.7B to 30B parameters, covering both dense and MoE architectures, and block sizes ranging from 4 to 64. We also show that with minimal additional supervised fine-tuning, block size can be increased from a base SDAR model at negligible cost. 5. Open-sourced models and inference engine. We release all trained SDAR models across different sizes(1.7B, 4B, 8B, 30B) and block configurations(from 4 to 64), along with both researchoriented and production-ready inference engines. For research and educational purposes, we provide JetEngine, a lightweight implementation prioritizing code clarity. For industrial applications, we offer a high-performance implementation on the lmdeploy framework with block-oriented optimizations for efficient SDAR inference. Our release also includes SDAR-30B-A3B-Sci, the most powerful diffusion-based reasoning model to date, and the first diffusion model capable of long chain-of-thought generation (see Appendix B), achieving state-of-the-art performance in complex reasoning tasks. Preliminary: Language Modeling Paradigm We begin by establishing our notation and reviewing the foundational paradigms in language modeling that are pertinent to our work. We consider a vocabulary V of size V. A token at position ℓ is represented by its index x ℓ ∈ V, and a sequence of length L is a tuple x = (x 1 , . . . , x L ). For computational purposes, we represent each token x ℓ by its one-hot vector x ℓ ∈ 0, 1 V . The entire sequence is thus represented by x = (x 1 , . . . , x L ). Finally, we denote the probability simplex over the vocabulary as ∆ V , and use Cat(•; p) for a categorical distribution with probabilities p ∈ ∆ V .