ACL Findings2026
Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders
Ailiang Lin, Zhuoyun Li, Keyu Mao, Kotaro Funakoshi, Manabu Okumura
Abstract
Large language models (LLMs) have been widely explored for embedding generation.While recent studies show that in-context learning (ICL) effectively enhances the representational capability of LLMs by prepending a few task-related demonstrations, it causes substantial token overhead due to the increased sequence length.In this work, we propose EPIC, a novel embedding-based in-context prompt training strategy that leverages ICL to generate high-quality embeddings while reducing computational burden during both training and inference.This approach replaces discrete text demonstrations with their corresponding continuous embeddings, which not only encourages the LLM to align semantically-related text pairs during contrastive learning, but also requires the model to interpret demonstration embeddings as part of the in-context prompt.Consequently, EPIC-trained models achieve excellent embedding performance both with or without in-context prompts at inference time.Comprehensive experiments demonstrate that our method establishes new state-of-the-art results on the MTEB benchmark, surpassing frontier models trained solely on publicly available retrieval data.Extensive ablation studies further validate the effectiveness and necessity of our mechanism.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on23
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
Related papers
- The Silent Amplifier: In-Context Examples Fuel Bias in Large Language ModelsXinwei Guo, Jiashi Gao, Junlei Zhou, Jiaxin Zhang et al.AAAI 2026
- Learning to Select In-Context Demonstration Preferred by Large Language ModelZheng Zhang, Shaocheng Lan, Lei Song, Jiang Bian et al.ACL Findings 2025 · 5 citations
- Training LLMs to be Better Text Embedders through Bidirectional ReconstructionChang Su, Dengliang Shi, Siyuan Huang, Jintao Du et al.EMNLP 2025
- Rethinking the Evaluation of In-Context Learning for LLMsGuoxin Yu, Lemao Liu, Mo Yu, Yue Yu et al.EMNLP 2024
- Compositional Exemplars for In-context LearningJiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu et al.ICML 2023 · 188 citations
