XShift: FPGA-efficient Binarized LLM with Joint Quantization and Sparsification

Binarization is a promising approach to significantly reduce computational complexity by replacing multiplications with hardwareefficient XNOR operations. However, the binarization of LLM activations often leads to severe accuracy degradation, while weight-only binarization fails to eliminate multipliers due to the Self-Attention mechanism. Furthermore, LLMs exhibit distinctive channel-level data distribution characteristics and differing computational and memory requirements between the Pre-fill and Decoding stages, necessitating a specialized inference framework. In response, we introduce XShift, an algorithm-hardware co-design framework optimized for efficient binarized LLM inference on FPGAs. XShift incorporates three key contributions: (1) a hardwarefriendly XNOR-Shift Encoding (XSE) format that transforms traditional multiplications into XNOR and shift operations, ensuring scalability and precision; (2) Hardware Adaptive Outlier and Sparsity (HAOS) techniques, which exploit channel-level data distribution and systolic array architectures for optimized quantization and sparsification; and (3) a dedicated hardware accelerator featuring an XNOR-Shift Systolic Array (XSSA) and an enhanced Base-2 SoftMax Converter (BSMC), designed to address the specific computational demands of binarized LLMs. Experimental evaluations on the Alveo U280 and U50 FPGA demonstrate that XShift achieves a $\mathbf{1 0 - 1 5 x}$ reduction in DSP resource usage while surpassing existing accelerators and GPUs in inference performance. Specifically, XShift delivers an average speedup of 4.174.76 x and a 14.29-6.95 x improvement in energy efficiency, alongside lower perplexity compared to other low-precision LLM techniques. These results underscore the potential of XShift for edge deployment of LLMs.