Fast and Accurate Causal Parallel Decoding using Jacobi Forcing

Multi-token generation has emerged as a promising paradigm for accelerating transformer-based large model inference. Recent efforts primarily explore diffusion Large Language Models (dLLMs) for parallel decoding to reduce inference latency. To achieve AR-level generation quality, many techniques adapt AR models into dLLMs to enable parallel decoding. However, they suffer from limited speedup compared to AR models due to a pretrain-to-posttrain mismatch. Specifically, the masked data distribution in post-training deviates significantly from the real-world data distribution seen during pretraining, and dLLMs rely on bidirectional attention, which conflicts with the causal prior learned during pretraining and hinders the integration of exact KV cache reuse. To address this, we introduce Jacobi Forcing, a progressive distillation paradigm where models are trained on their own generated parallel decoding trajectories, smoothly shifting AR models into efficient parallel decoders while preserving their pretrained causal inference property. The models trained under this paradigm, Jacobi Forcing Model, achieves 3.8× wall-clock speedup on coding benchmarks with minimal loss in performance. Based on Jacobi Forcing Model's trajectory characteristics, we introduce multi-block decoding with rejection recycling, which enables up to 4.5× higher token acceptance count per iteration and nearly 4.0× wall-clock speedup, effectively trading additional compute for lower inference latency. Our code is available at https://github.com/hao-ai-lab/JacobiForcing . INTRODUCTION Modern large language models (LLMs), such as GPT-5 (OpenAI, 2025), Gemini-2.5 (DeepMind, 2025), and Kimi-K2 (Team et al., 2025) , excel at complex and interactive agentic tasks. Yet, autoregressive (AR) decoding generates tokens sequentially, limiting parallelism and leading to high latency. To address this, recent work explores predicting multiple future tokens natively in transformer-based models without relying on auxiliary draft models. A popular approach is diffusion-based language models (dLLMs), which relax left-to-right generation by modeling the entire sequence jointly and decoding via full-sequence denoising (Nisonoff et al., 2024; Schiff et al., 2024; Inception Labs, 2025). This, in turn, enables highly parallelizable computation. However, open pretrained dLLMs (Ye et al., 2025; Zhu et al., 2025; Nie et al., 2025a) underperform AR models in generation quality, mainly due to their negative evidence lower bound (NELBO) training objective, a loose bound on AR's negative log-likelihood (NLL) that is proven less efficient (Cheng et al., 2025; Nie et al., 2024; Arriola et al., 2025) . To preserve the generation quality of frontier AR models, the community has adapted high-quality AR models into dLLMs for parallel decoding (JetAstra, 2025; Wu et al., 2025b). Concretely, they perform block-wise perturbations of pretrained data by randomly masking tokens following the dLLMs recipe, and leverage these data to posttrain AR models by modifying the attention mask to enable block-wise bidirectional attention and replacing the training objective from NLL to NELBO. This adaptation delivers limited speedup under quality constraints, primarily due to a significant pretrain-to-posttrain mismatch. Specifically, enforcing block-wise bidirectional attention conflicts * Equal contributions. Part of work was done during Lanxiang's internship at Snowflake.