Unlocking Full Efficiency of Token Filtering in Large Language Model Training

Token filtering has been proposed to enhance the utility of large language models (LLMs) by eliminating inconsequential tokens during training. While using fewer tokens is expected to reduce computational workloads, existing methods have not yet achieved a real-world efficiency boost. This is primarily due to two factors: (1) existing work has inadequate sparsity for speedup, and (2) token filtering operates within a sparsity range that is non-standard in existing machine learning (ML) libraries and thus cannot be efficiently supported. This paper presents CENTRIFUGE 1 , a system that leverages algorithm and system co-design to unleash the full efficiency of token filtering in LLM training. At the algorithm level, CENTRIFUGE filters activations of inconsequential tokens in the attention backward kernel to amplify the sparsity in backward computation. At the system level, CENTRIFUGE proposes an automatic workflow that transforms sparse GEMM into dimension-reduced dense GEMM for optimized efficiency using standard ML libraries. Evaluations on models with various scales-from 1.1B to 40B-demonstrate that CENTRIFUGE reduces backpropagation time by up to 49.9% and end-to-end training time by up to 34.7% when filtering 50% of tokens. Utility assessments indicate that CENTRIFUGE preserves the utility benefits of token filtering and significantly enhances model performance by up to 26.6% compared to standard training. CENTRIFUGE is designed for seamless integration into existing LLM training frameworks, enabling systems already utilizing token filtering to accelerate training with just one line of code. * Junxue Zhang and Kai Chen are the corresponding authors. 1 A centrifuge is a modern system used in chemistry laboratories to efficiently filter different elements.