Lune

ICLR2024Top-tier venue

In-context Autoencoder for Context Compression in a Large Language Model

Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, Furu Wei

Abstract

We propose the In-context Autoencoder (ICAE), leveraging the power of a large language model (LLM) to compress a long context into short compact memory slots that can be directly conditioned on by the LLM for various purposes. ICAE is first pretrained using both autoencoding and language modeling objectives on massive text data, enabling it to generate memory slots that accurately and comprehensively represent the original context. Then, it is fine-tuned on instruction data for producing desirable responses to various prompts. Experiments demonstrate that our lightweight ICAE, introducing about 1% additional parameters, effectively achieves 4× context compression based on Llama, offering advantages in both improved latency and GPU memory cost during inference, and showing an interesting insight in memorization as well as potential for scalability. These promising results imply a novel perspective on the connection between working memory in cognitive science and representation learning in LLMs, revealing ICAE's significant implications in addressing the long context problem and suggesting further research in LLM context management. Our data, code and models are available at https://github.com/getao/icae . As ar&ficial intelligence becomes an increasingly powerful force, some of the world's biggest companies are worrying about how the technology will be used ethically, and how the public will perceive its spread. To combat these problems (among others), five tech companies -Google, Amazon, MicrosoD, Facebook, and IBM -set up a research group called the Partnership on AI. … AI taking white collar jobs, eroding trust in public media, becoming embedded in public ins&tu&ons like the courts and hospitals: these are the sorts of problems facing the industry in the future.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

63 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving cites LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
  • CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning cites Finetuned Language Models are Zero-Shot Learners
  • The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning cites Training language models to follow instructions with human feedback
  • Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning cites Finetuned Language Models are Zero-Shot Learners
  • Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning cites LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
  • Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Finetuned Language Models are Zero-Shot Learners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Finetuned Language Models are Zero-Shot Learners cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites Finetuned Language Models are Zero-Shot Learners
  • LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers63

  1. The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context LearningBill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri et al.ICLR 2024 · 299 citations
  2. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model ServingYuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray et al.SIGCOMM 2024 · 111 citations
  3. TabPedia: Towards Comprehensive Visual Table Understanding with Concept SynergyWeichao Zhao, Hao Feng, Qi Liu, Jingqun Tang et al.NeurIPS 2024 · 97 citations
  4. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language ModelsHuiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang et al.EMNLP 2023 · 94 citations
  5. Multimodal Task Vectors Enable Many-Shot Multimodal In-Context LearningBrandon Huang, Chancharik Mitra, Leonid Karlinsky, Assaf Arbelle et al.NeurIPS 2024 · 60 citations

Builds on19

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  3. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  4. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  5. Compressive Transformers for Long-Range Sequence ModellingJack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier et al.ICLR 2020 · 833 citations

Related papers

  1. GMSA: Enhancing Context Compression via Group Merging and Layer Semantic AlignmentJiwei Tang, Zhicheng Zhang, Shunlong Wu, Jingheng Ye et al.ACL 2026 · 24 citations
  2. Pretraining Context Compressor for Large Language Models with Embedding-Based MemoryYuhong Dai, Jianxun Lian, Yitian Huang, Wei Zhang et al.ACL 2025
  3. LLoCO: Learning Long Contexts OfflineSijun Tan, Xiuyu Li, Shishir G. Patil, Ziyang Wu et al.EMNLP 2024 · 3 citations
  4. E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and ReasoningZihan Liao, Jun Wang, Hang Yu, Lingxiao Wei et al.EMNLP 2025
  5. Learning to Compress: Unlocking the Potential of Large Language Models for Text RepresentationYeqin Zhang, Yizheng Zhao, Chen Hu, Binxing Jiao et al.AAAI 2026 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines