LLMFolder: Revisiting Constant Folding in Large Language Models

Large language models (LLMs) demonstrate remarkable capabilities but face deployment challenges due to their massive parameter counts. While pruning can reduce model size, it leads to significant accuracy degradation under high compression ratios. We present a novel perspective inspired by constant folding in compiler optimization. Our approach enables parameter reduction by treating activation functions in LLMs as linear functions.

However, recent LLMs use complex non-linear activations like GELU that prevent direct application of this technique. We propose LLMFolder, which enables optimization of LLMs with non-linear activations by partially approximating them with linear functions in frequently occurring input ranges. For outlier inputs, LLMFolder employs an online predictor to dynamically fall back to original computations.

Our experiments demonstrate that LLMFolder achieves 80% parameter reduction in feed-forward networks, while significantly outperforming state-of-the-art pruning methods with up to 65% higher accuracy. When combined with quantization and pruning, LLMFolder can further achieve 92.5% parameter reduction with only 4.4% average accuracy drop for a 7B model, where neither technique alone or combination can achieve. In practical deployments for