Oltron: Algorithm-Hardware Co-design for Outlier-Aware Quantization of LLMs with Inter-/Intra-Layer Adaptation

In Large Language Models (LLMs), outliers are identified by a small number of values with exceptionally high magnitudes, critically affecting model accuracy. Researchers have proposed several mixed-precision quantization techniques to manage these activation outliers. These approaches, employing value-wise outlier granularity, face challenges in balancing model accuracy with hardware efficiency. To address this issue, we capitalize on the observation that activation outliers of LLMs typically cluster within specific channels. Consequently, we introduce Oltron, a comprehensive software/hardware co-design strategy for outlier-aware quantization of LLMs with inter-/intra-layer adaptation. Our method includes three key innovations: firstly, a novel quantization algorithm that identifies the optimal ratio of outliers across different layers and channel groups within a layer; secondly, a reconfigurable architecture that adapts to inter- and intra-layer distributions; and thirdly, a tile-based dataflow optimizer that intricately arranges complex computations and memory access for mixed-precision tensors. Oltron outperforms the state-of-the-art outlier-aware accelerator, OliVe, achieving a 1.9x performance boost and 1.6x greater energy efficiency, while also enhancing model accuracy.