Lune

ACL Findings2026

SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation

Shuang Cheng, Yihan Bian, Dawei Liu, Yuhua Jiang, Yihao Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, Kai Chen, Biqing Qi, Bowen Zhou

Abstract

Autoregressive (AR) language modeling remains the dominant paradigm due to its dense supervision signal and highly optimized serving infrastructure, but its strictly causal, tokenby-token decoding limits parallelism and noncausal modeling. While masked diffusion offers a promising path toward parallel generation, it faces two critical bottlenecks: training inefficiency stemming from sparse masked objectives, and high latency caused by iterative whole-sequence denoising. We present a systematic study of blockwise discrete diffusion, a pragmatic middle ground that preserves AR-compatible serving while enabling parallel intra-block generation. Our study proceeds in four steps: (i) a controlled, compute-and scale-matched comparison revealing that AR is a more effective backbone for blockwise hybrids than masked diffusion objectives; (ii) a scalable conversion recipe, SDAR, validating that AR models spanning 1.7B to 30B parameters can be adapted into block diffusion models with minimal compute while preserving backbone capabilities; and (iii) a systematic characterization of decoding dynamics, which reveals a virtuous cycle where larger models enable more aggressive parallel decoding, achieving theoretical speedups over 5× and wallclock speedups of 2.3× on H200 GPUs in latency-critical regimes; and (iv) an investigation of local non-causal modeling capabilities, showing that SDAR's local bidirectional attention overcomes causal bottlenecks in scientific domains (e.g., chemistry) and enables robust test-time scaling. We release the full model suite, the training framework, and our inference engines for further innovation in nonautoregressive generative paradigms.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

18 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models cites Let's Verify Step by Step
  • Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
  • Fast-dLLM v2: Efficient Block-Diffusion LLM cites Language Models are Few-Shot Learners
  • Efficient Memory Management for Large Language Model Serving with PagedAttention cites Language Models are Few-Shot Learners
  • d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation cites Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models
  • d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation cites Fast-dLLM v2: Efficient Block-Diffusion LLM
  • d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation cites Let's Verify Step by Step
  • AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Size cites Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models
  • Language models enable zero-shot prediction of the effects of mutations on protein function cites Language Models are Few-Shot Learners
  • WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference cites Measuring Massive Multitask Language Understanding
  • WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference cites Efficient Memory Management for Large Language Model Serving with PagedAttention
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers18

  1. Fast-dLLM v2: Efficient Block-Diffusion LLMChengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao et al.ICLR 2026 · 132 citations
  2. Revolutionizing Reinforcement Learning Framework for Diffusion Large Language ModelsYinjie Wang, Ling Yang, Bowen Li, Ye Tian et al.ICLR 2026 · 79 citations
  3. WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast InferenceAiwei Liu, Minghua He, Shaoxun Zeng, Sijun Zhang et al.ICML 2026 · 37 citations
  4. d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory DistillationYu-Yang Qian, Junda Su, Lanxiang Hu, Peiyuan Zhang et al.ICML 2026 · 33 citations
  5. AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block SizeGuanxi Lu, Hao Mark Chen, Yuto Karashima, Zhican Wang et al.ICLR 2026 · 32 citations

Builds on20

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  3. Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
  4. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
  5. Language models enable zero-shot prediction of the effects of mutations on protein functionJoshua Meier, Roshan Rao, Robert Verkuil, Jason Liu et al.NeurIPS 2021 · 969 citations

Related papers

  1. DiffuMamba: High-Throughput Diffusion LMs with Mamba BackboneVaibhav Singh, Oleksiy Ostapenko, Pierre-André Noël, Eugene Belilovsky et al.ICML 2026
  2. Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in SpeedYonggan Fu, Lexington Whalen, Zhifan Ye, Xin Dong et al.ICML 2026 · 22 citations
  3. SDAR-VL: Stable and Efficient Block-wise Diffusion for Vision-Language UnderstandingShuang Cheng, Yuhua Jiang, Zineng Zhou, Dawei Liu et al.ACL 2026 · 8 citations
  4. Beyond Autoregression: Fast LLMs via Self-Distillation Through TimeJustin Deschenaux, Caglar GulcehreICLR 2025
  5. Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion ForcingXu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin et al.ICLR 2026 · 116 citations
Dusk over the sea between two cliffs drawn in fine vertical lines