Streamlining Redundant Layers to Compress Large Language Models

This paper introduces LLM-Streamline, a pioneer work on layer pruning for large language models (LLMs). It is based on the observation that different layers have varying impacts on hidden states, enabling the identification of less important layers to be pruned. LLM-Streamline comprises two parts: layer pruning, which removes consecutive layers with the lowest importance based on target sparsity, and layer replacement, a novel module that trains a lightweight network to replace the pruned layers to mitigate performance loss. Additionally, a new metric called stability is proposed to address the limitations of the widely used accuracy metric in evaluating model compression. Experiments show that LLM-Streamline outperforms both previous and concurrent state-of-the-art pruning methods in terms of both performance and training efficiency. Our code is available at this repository. Layer + -… Layer -Layer … Layer 1 INTRODUCTION Large language models (LLMs) built on the Transformer architecture (Vaswani et al., 2017) have gained widespread attention and are applied across diverse domains and tasks. However, as LLMs increase in size, their hardware requirements escalate substantially, thereby constraining their applicability and impeding their deployment in real-world scenarios. To reduce the hardware requirements † These authors contributed equally. * Jing Zhang is the corresponding author.