OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference

Recent studies have revealed the sink token, which receives disproportionately high attention despite its limited semantic role. In this paper, we first explore the relationship between the sink token and other tokens beyond attention, by analyzing their similarity in hidden states. We observe that as layers deepen, the cosine similarity between the normalized hidden states of the sink token and those of other tokens increases, and that the normalized hidden states of the sink token exhibit negligible changes. These imply that other tokens are consistently directed toward the sink token throughout the layers. Next, we propose a dynamic token selection method, called OrthoRank, using these findings to select important tokens. Specifically, in a certain layer, we define token importance by the speed at which the token moves toward the sink token. This is converted into orthogonality with the sink token, meaning that tokens that are more orthogonal to the sink token are assigned greater importance. Extensive experiments show that our method results in lower perplexity and higher zero-shot accuracy compared to layer pruning methods at the same sparsity ratio with comparable throughput, while also outperforming on LongBench.