ACL Findings2026
Training-Free Adaptive Speculative Decoding via Linguistic Priors
Jingyi Wang, Jiaqi Huang, Zunnan Xu, Jun Zhou, Kehong Yuan, Xiang Qian
Abstract
Speculative decoding (SPD) has emerged as a promising technique to accelerate Large Language Model (LLM) inference. However, current approaches typically enforce a uniform verification standard, neglecting the inherent heterogeneity of natural language and failing to distinguish between semantically-rich content and structurally-predictable syntax. In this paper, we propose LinguaSpec, a training-free framework that leverages linguistic priors to enable adaptive drafting and verification. Specifically, we introduce: (1) a Static Linguistic Probe (SLP) to categorize tokens with zero latency; (2) Syntactic Normalized Surprisal (SNS) to calibrate uncertainty against categoryspecific entropy; and (3) a dual strategy of Syntactically-Guided Elastic Expansion and POS-Adaptive Deferred Verification to dynamically adjust drafting depth and verification rigor. By balancing semantic integrity with structural efficiency, LinguaSpec significantly accelerates inference without requiring additional training. Experimental results demonstrate its superior performance across diverse benchmarks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Break the Sequential Dependency of LLM Inference Using Lookahead Decoding cites Fast Inference from Transformers via Speculative Decoding
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test cites Fast Inference from Transformers via Speculative Decoding
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty cites Fast Inference from Transformers via Speculative Decoding
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites Fast Inference from Transformers via Speculative Decoding
Builds on18
- Fast Inference from Transformers via Speculative DecodingYaniv Leviathan, Matan Kalman, Yossi MatiasICML 2023 · 1,472 citations
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsTianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng et al.ICML 2024 · 669 citations
- EAGLE: Speculative Sampling Requires Rethinking Feature UncertaintyYuhui Li, Fangyun Wei, Chao Zhang, Hongyang ZhangICML 2024 · 424 citations
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time TestYuhui Li, Fangyun Wei, Chao Zhang, Hongyang ZhangNeurIPS 2025 · 347 citations
- Break the Sequential Dependency of LLM Inference Using Lookahead DecodingYichao Fu, Peter Bailis, Ion Stoica, Hao ZhangICML 2024 · 290 citations
Related papers
- UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and HardwareTruong Dinh Do, Nguyen-Khang Le, Le-Minh NguyenACL 2026
- RACER: Retrieval-Augmented Contextual Rapid Speculative DecodingZihong Zhang, Zuchao Li, Lefei Zhang, Ping Wang et al.ACL Findings 2026 · 1 citation
- Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact MatchJinze Li, Yixing Xu, Guanchen Li, Shuo Yang et al.ICLR 2026 · 12 citations
- LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token SpeculationTianyu Liu, Qitan Lv, Hao Li, Xing Gao et al.ACL Findings 2026
- Block Verification Accelerates Speculative DecodingZiteng Sun, Uri Mendlovic, Yaniv Leviathan, Asaf Aharoni et al.ICLR 2025
