UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware

Speculative decoding accelerates large language model (LLM) inference through a draftand-verify paradigm, yet existing methods face three key limitations: reliance on fixed draft templates that ignore device-specific verification costs, lack of mechanisms to assess draft token quality, and suboptimal tree expansion strategies. We introduce UNISPEC, a trainingfree, lossless speculative decoding framework that enables robust, plug-and-play LLM acceleration across diverse hardware configurations and languages. UNISPEC incorporates three novel components: (1) a device-aware calibration mechanism that determines the optimal draft size by measuring the acceptancetime trade-off on each target device; (2) a confidence score estimation module that assigns quality scores to n-grams based on the verifier's token probabilities, enabling selective retention of high-quality draft candidates; and (3) an improved tree expansion strategy that broadens first-level exploration and applies threshold-based filtering to prune lowconfidence nodes. To comprehensively evaluate multilingual performance, we create a comprehensive benchmark, covering seven languages across seven generation tasks. Experiments with various LLM architectures, hardware environments, and languages demonstrate that UNISPEC consistently outperforms existing training-free methods, achieving speedups of up to 2.6× while maintaining output quality identical to standard autoregressive decoding. Our code and benchmark are publicly available.