Lune

ACL Findings2025

Understanding the Influence of Synthetic Data for Text Embedders

Jacob Mitchell Springer, Vaibhav Adlakha, Siva Reddy, Aditi Raghunathan, Marius Mosbach

Abstract

Recent progress in developing general purpose text embedders has been driven by training on ever-growing corpora of synthetic LLMgenerated data. Nonetheless, no publicly available synthetic dataset exists, posing a barrier to studying its role for generalization. To address this issue, we first reproduce and publicly release the synthetic data proposed by Wang et al. (2024) (Mistral-E5). Our synthetic data is high quality and leads to consistent improvements in performance. Next, we critically examine where exactly synthetic data improves model generalization. Our analysis reveals that benefits from synthetic data are sparse and highly localized to individual datasets. Moreover, we observe trade-offs between the performance on different categories and data that benefits one task, degrades performance on another. Our findings highlight the limitations of current synthetic data approaches for building generalpurpose embedders and challenge the notion that training on synthetic data leads to more robust embedding models across tasks.

jakespringer/open-synthetic-embeddings jspringer/open-synthetic-embeddings

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
  • Promptagator: Few-shot Dense Retrieval From 8 Examples cites Dense Passage Retrieval for Open-Domain Question Answering
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language ModelsTobias Grantner, Emanuel Sallinger, Martin FlechlACL 2026

Builds on10

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis et al.EMNLP 2020 · 142 citations
  4. TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding LearningKexin Wang, Nils Reimers, Iryna GurevychEMNLP Findings 2021 · 130 citations
  5. Promptagator: Few-shot Dense Retrieval From 8 ExamplesZhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan et al.ICLR 2023 · 46 citations

Related papers

  1. Improving Text Embeddings with Large Language ModelsLiang Wang, Nan Yang, Xiaolong Huang, Linjun Yang et al.ACL 2024
  2. mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic DataHaonan Chen, Liang Wang, Nan Yang, Yutao Zhu et al.ACL Findings 2025 · 6 citations
  3. FinMTEB: Finance Massive Text Embedding BenchmarkYixuan Tang, Yi YangEMNLP 2025 · 4 citations
  4. A synthetic data approach for domain generalization of NLI modelsMohammad Javad Hosseini, Andrey Petrov, Alex Fabrikant, Annie LouisACL 2024 · 3 citations
  5. Meta-Task Prompting Elicits Embeddings from Large Language ModelsYibin Lei, Di Wu, Tianyi Zhou, Tao Shen et al.ACL 2024 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines