EMNLP Findings2024
Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability
Tsz Ting Chung, Leyang Cui, Lemao Liu, Xinting Huang, Shuming Shi, Dit-Yan Yeung
Abstract
Large Language Models (LLMs) have demonstrated impressive capabilities in a wide range of natural language processing tasks when leveraging in-context learning. To mitigate the additional computational and financial costs associated with in-context learning, several prompt compression methods have been proposed to compress the in-context learning prompts. Despite their success, these methods face challenges with transferability due to model-specific compression, or rely on external training data, such as GPT-4. In this paper, we investigate the ability of LLMs to develop a unified compression method that discretizes uninformative tokens, utilizing a self-supervised pre-training technique. By introducing a small number of parameters during the continual pretraining, the proposed Selection-p produces a probability for each input token, indicating whether to preserve or discard it. Experiments show Selection-p achieves state-of-the-art performance across numerous classification tasks, achieving compression rates of up to 10 times while experiencing only a marginal 0.8% decrease in performance. Moreover, it exhibits superior transferability to different models compared to prior work. Additionally, we further analyze how Selection-p helps maintain performance on in-context learning with long contexts.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
- Learning to Compress Prompts with Gist Tokens cites LoRA: Low-Rank Adaptation of Large Language Models
- Learning to Compress Prompts with Gist Tokens cites Language Models are Few-Shot Learners
- Learning to Compress Prompts with Gist Tokens cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Many-Shot CoT-ICL: Making In-Context Learning Truly Learn cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers1
Builds on21
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- DynamicViT: Efficient Vision Transformers with Dynamic Token SparsificationYongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu et al.NeurIPS 2021 · 1,343 citations
- Learning to Compress Prompts with Gist TokensJesse Mu, Xiang Li, Noah D. GoodmanNeurIPS 2023 · 488 citations
Related papers
- TACO-RL: Task Aware Prompt Compression Optimization with Reinforcement LearningShivam Shandilya, Menglin Xia, Supriyo Ghosh, Huiqiang Jiang et al.ACL Findings 2025 · 9 citations
- From Reading to Compressing: Exploring the Multi-document Reader for Prompt CompressionEunseong Choi, Sunkyung Lee, Minjin Choi, June Park et al.EMNLP Findings 2024
- 500xCompressor: Generalized Prompt Compression for Large Language ModelsZongqian Li, Yixuan Su, Nigel CollierACL 2025 · 35 citations
- Learning to Compress: Unlocking the Potential of Large Language Models for Text RepresentationYeqin Zhang, Yizheng Zhao, Chen Hu, Binxing Jiao et al.AAAI 2026 · 2 citations
- Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM InferenceBarys Liskavets, Maxim Ushakov, Shuvendu Roy, Mark Klibanov et al.AAAI 2025 · 41 citations
