Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decoding

Speculative decoding has emerged as a promising lossless approach to accelerating Large Language Models (LLMs). Recently, as the overhead of the decode stage has increased in reasoning LLMs, while algorithmic approximations cause significant accuracy degradation, lossless acceleration through speculative decoding has become a key component for efficient LLM serving systems. However, despite numerous advances, a speculative decoding method that can provide sufficient performance at low batch sizes without requiring additional training remains elusive. This presents a considerable challenge for achieving lossless acceleration in low-batch LLM inference on consumer-grade devices. To address this limitation, we propose Cassandra, an algorithm-hardware co-designed self-speculative decoding framework tailored explicitly for low-batch scenarios. The core idea of Cassandra is to construct a high-performance, training-free draft model through fine-grained data selection. By leveraging optimized pruning and mantissa truncation, Cassandra identifies and isolates the most salient values within both model weights and the Key-Value (KV) cache. These selected values are first loaded to rapidly generate candidate tokens, followed by parallel verification using full-precision data. This design achieves substantial performance gains over prior self-speculative decoding methods, which primarily rely on layer skipping or structured KV cache compression. In addition, to mitigate the overhead of format conversion between the Cassandra representation and standard floating-point formats, we introduce a lightweight encoder-decoder hardware module designed for seamless integration with commercial GPUs and NPUs. Our evaluation demonstrates that Cassandra achieves a performance improvement of up to $2.41 \times$ than the BFloat16 baseline without additional training. Moreover, when running Llama3-8B on an RTX 4090, Cassandra is capable of generating $1.81 \times$ more tokens under a fixed memory budget compared to Eagle-3, a state-of-the-art speculative decoding method.