ACL Findings2025
Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings
Abstract
Prompt-based text embedding models, which generate task-specific embeddings upon receiving tailored prompts, have recently demonstrated remarkable performance.However, their resulting embeddings often have thousands of dimensions, leading to high storage costs and increased computational costs of embedding-based operations.In this paper, we investigate how post-hoc dimensionality reduction applied to the embeddings affects the performance of various tasks that leverage these embeddings, specifically classification, clustering, retrieval, and semantic textual similarity (STS) tasks.Our experiments show that even a naive dimensionality reduction, which keeps only the first 25% of the dimensions of the embeddings, results in a very slight performance degradation, indicating that these embeddings are highly redundant.Notably, for classification and clustering, even when embeddings are reduced to less than 0.5% of the original dimensionality the performance degradation is very small.To quantitatively analyze this redundancy, we perform an analysis based on the intrinsic dimensionality and isotropy of the embeddings.Our analysis reveals that embeddings for classification and clustering, which are considered to have very high dimensional redundancy, exhibit lower intrinsic dimensionality and less isotropy compared with those for retrieval and STS. Robust to ReductionIntrinsic Dimension: Low Isotropy: Low 2 Avg.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- TermGPT: Multi-Level Contrastive Fine-Tuning for Terminology Adaptation in Legal and Financial Domains cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
- Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks cites Matryoshka Representation Learning
- Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
- SimCSE: Simple Contrastive Learning of Sentence Embeddings cites On the Sentence Embeddings from Pre-trained Language Models
- Provable Accuracy Collapse of Embedding-Based Representations under Dimensionality Mismatch cites Matryoshka Representation Learning
- Provable Accuracy Collapse of Embedding-Based Representations under Dimensionality Mismatch cites Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks
Cited by top-tier papers4
- TermGPT: Multi-Level Contrastive Fine-Tuning for Terminology Adaptation in Legal and Financial DomainsYidan Sun, Mengying Zhu, Feiyue Chen, Yangyang Wu et al.AAAI 2026 · 1 citation
- JHQ: Johnson-Lindenstrauss Enhanced Hierarchical Quantization for High-Dimensional Approximate Nearest Neighbor SearchJiabao Han, Mengxuan Zhang, Goce TrajcevskiVLDB 2026 · 1 citation
- Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification TasksSotaro Takeshita, Yurina Takeshita, Daniel Ruffinelli, Simone Paolo PonzettoEMNLP 2025
- Provable Accuracy Collapse of Embedding-Based Representations under Dimensionality MismatchDionysis Arvanitakis, Vaggos Chatziafratis, Yiyuan LuoICML 2026
Builds on27
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- On the Sentence Embeddings from Pre-trained Language ModelsBohan Li, Hao Zhou, Junxian He, Mingxuan Wang et al.EMNLP 2020 · 538 citations
- Matryoshka Representation LearningAditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford et al.NeurIPS 2022 · 364 citations
Related papers
- A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key TokensZhijie Nie, Richong Zhang, Zhanyu WuACL 2025 · 5 citations
- Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding DimensionsJinsung Yoon, Rajarishi Sinha, Sercan Ömer Arik, Tomas PfisterEMNLP 2024 · 1 citation
- Scaling Laws for Embedding Dimension in Information RetrievalJulian Killingback, Mahta Rafiee, Madine Manas, Hamed ZamaniSIGIR 2026
- One Embedder, Any Task: Instruction-Finetuned Text EmbeddingsHongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang et al.ACL Findings 2023 · 134 citations
- Measuring the Intrinsic Dimension of Earth RepresentationsArjun Rao, Marc Rußwurm, Konstantin Klemmer, Esther RolfICLR 2026 · 13 citations
