ACL Findings2025
Understanding the Influence of Synthetic Data for Text Embedders
Jacob Mitchell Springer, Vaibhav Adlakha, Siva Reddy, Aditi Raghunathan, Marius Mosbach
Abstract
Recent progress in developing general purpose text embedders has been driven by training on ever-growing corpora of synthetic LLMgenerated data. Nonetheless, no publicly available synthetic dataset exists, posing a barrier to studying its role for generalization. To address this issue, we first reproduce and publicly release the synthetic data proposed by Wang et al. (2024) (Mistral-E5). Our synthetic data is high quality and leads to consistent improvements in performance. Next, we critically examine where exactly synthetic data improves model generalization. Our analysis reveals that benefits from synthetic data are sparse and highly localized to individual datasets. Moreover, we observe trade-offs between the performance on different categories and data that benefits one task, degrades performance on another. Our findings highlight the limitations of current synthetic data approaches for building generalpurpose embedders and challenge the notion that training on synthetic data leads to more robust embedding models across tasks.
jakespringer/open-synthetic-embeddings jspringer/open-synthetic-embeddings
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
- Promptagator: Few-shot Dense Retrieval From 8 Examples cites Dense Passage Retrieval for Open-Domain Question Answering
Cited by top-tier papers1
Builds on10
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis et al.EMNLP 2020 · 142 citations
- TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding LearningKexin Wang, Nils Reimers, Iryna GurevychEMNLP Findings 2021 · 130 citations
- Promptagator: Few-shot Dense Retrieval From 8 ExamplesZhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan et al.ICLR 2023 · 46 citations
Related papers
- Improving Text Embeddings with Large Language ModelsLiang Wang, Nan Yang, Xiaolong Huang, Linjun Yang et al.ACL 2024
- mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic DataHaonan Chen, Liang Wang, Nan Yang, Yutao Zhu et al.ACL Findings 2025 · 6 citations
- FinMTEB: Finance Massive Text Embedding BenchmarkYixuan Tang, Yi YangEMNLP 2025 · 4 citations
- A synthetic data approach for domain generalization of NLI modelsMohammad Javad Hosseini, Andrey Petrov, Alex Fabrikant, Annie LouisACL 2024 · 3 citations
- Meta-Task Prompting Elicits Embeddings from Large Language ModelsYibin Lei, Di Wu, Tianyi Zhou, Tao Shen et al.ACL 2024 · 6 citations
