Lune

ACL Findings2024

Anchor-based Large Language Models

Jianhui Pang, Fanghua Ye, Derek F. Wong, Xin He, Wanshun Chen, Longyue Wang

Abstract

Large language models (LLMs) predominantly employ decoder-only transformer architectures, necessitating the retention of keys/values information for historical tokens to provide contextual information and avoid redundant computation. However, the substantial size and parameter volume of these LLMs require massive GPU memory. This memory demand increases with the length of the input text, leading to an urgent need for more efficient methods of information storage and processing. This study introduces Anchor-based LLMs (AnLLMs), which utilize an innovative anchor-based self-attention network (AnSAN) and also an anchor-based inference strategy. This approach enables LLMs to compress sequence information into an anchor token, reducing the keys/values cache and enhancing inference efficiency. Experiments on question-answering benchmarks reveal that An-LLMs maintain similar accuracy levels while achieving up to 99% keys/values cache reduction and up to 3.5 times faster inference. Despite a minor compromise in accuracy, the substantial enhancements of AnLLMs employing the AnSAN technique in resource utilization and computational efficiency underscore their potential for practical LLM applications. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

9 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LightThinker: Thinking Step-by-Step Compression cites Learning to Compress Prompts with Gist Tokens
  • Learning to Compress Prompts with Gist Tokens cites Compressive Transformers for Long-Range Sequence Modelling
  • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness cites Compressive Transformers for Long-Range Sequence Modelling
  • Efficient Memory Management for Large Language Model Serving with PagedAttention cites FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  • InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models cites LightThinker: Thinking Step-by-Step Compression
  • InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models cites Efficient Memory Management for Large Language Model Serving with PagedAttention
  • EARN: Efficient Inference Acceleration for LLM-based Generative Recommendation by Register Tokens cites Learning to Compress Prompts with Gist Tokens
  • KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding cites Efficient Memory Management for Large Language Model Serving with PagedAttention
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers9

  1. KVLink: Accelerating Large Language Models via Efficient KV Cache ReuseJingbo Yang, Bairu Hou, Wei Wei, Yujia Bao et al.NeurIPS 2025 · 83 citations
  2. InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language ModelsYuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang et al.ICLR 2026 · 48 citations
  3. KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional EmbeddingLuohe Shi, Zuchao Li, Lefei Zhang, Baoyuan Qi et al.ACL 2025 · 5 citations
  4. LightThinker: Thinking Step-by-Step CompressionJintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo et al.EMNLP 2025 · 2 citations
  5. EARN: Efficient Inference Acceleration for LLM-based Generative Recommendation by Register TokensChaoqun Yang, Xinyu Lin, Wenjie Wang, Yongqi Li et al.KDD 2025 · 1 citation

Builds on15

  1. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
  2. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  3. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
  4. Compressive Transformers for Long-Range Sequence ModellingJack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier et al.ICLR 2020 · 833 citations
  5. Learning to Compress Prompts with Gist TokensJesse Mu, Xiang Li, Noah D. GoodmanNeurIPS 2023 · 488 citations

Related papers

  1. Autoencoding-Free Context Compression for LLMs via Contextual Semantic AnchorsXin Liu, Runsong Zhao, Pengcheng Huang, Xinyu Liu et al.ICLR 2026 · 16 citations
  2. KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial RecomputationChaoyi Jiang, Lei Gao, Hossein Entezari Zarch, Murali AnnavaramACL Findings 2025 · 4 citations
  3. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM InferenceDongjie Yang, Xiaodong Han, Yan Gao, Yao Hu et al.ACL Findings 2024 · 31 citations
  4. Inference-Time Hyper-Scaling with KV Cache CompressionAdrian Lancucki, Konrad Staniszewski, Piotr Nawrot, Edoardo Maria PontiNeurIPS 2025 · 36 citations
  5. Latent-Condensed Transformer for Efficient Long Context ModelingZeng You, Yaofo Chen, Qiuwu Chen, Ying Sun et al.ACL 2026
Dusk over the sea between two cliffs drawn in fine vertical lines