Lune

EMNLP Findings2024

Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domains

Krithika Ramesh, Nupoor Gandhi, Pulkit Madaan, Lisa Bauer, Charith Peris, Anjalie Field

Abstract

The difficulty of anonymizing text data hinders the development and deployment of NLP in high-stakes domains that involve private data, such as healthcare and social services. Poorly anonymized sensitive data cannot be easily shared with annotators or external researchers, nor can it be used to train public models. In this work, we explore the feasibility of using synthetic data generated from differentially private language models in place of real data to facilitate the development of NLP in these domains without compromising privacy. In contrast to prior work, we generate synthetic data for real high-stakes domains, and we propose and conduct use-inspired evaluations to assess data quality. Our results show that prior simplistic evaluations have failed to highlight utility, privacy, and fairness issues in the synthetic data. Overall, our work underscores the need for further improvements to synthetic data generation for it to be a viable way to enable privacy-preserving data sharing.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Probabilistic Reasoning with LLMs for Privacy Risk Estimation cites LoRA: Low-Rank Adaptation of Large Language Models
  • Controlled Generation for Private Synthetic Text cites Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
  • Controlled Generation for Private Synthetic Text cites Large Language Models Can Be Strong Differentially Private Learners
  • Controlled Generation for Private Synthetic Text cites Deep Learning with Differential Privacy
  • Large Language Models Can Be Strong Differentially Private Learners cites Deep Learning with Differential Privacy
  • Large Language Models Can Be Strong Differentially Private Learners cites The Curious Case of Neural Text Degeneration
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Probabilistic Reasoning with LLMs for Privacy Risk EstimationJonathan Zheng, Alan Ritter, Sauvik Das, Wei (Coco) XuNeurIPS 2025 · 3 citations
  2. Controlled Generation for Private Synthetic TextZihao Zhao, Anjalie FieldEMNLP 2025

Builds on13

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Deep Learning with Differential PrivacyMartín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan et al.CCS 2016 · 7,620 citations
  3. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  4. Large Language Models Can Be Strong Differentially Private LearnersXuechen Li, Florian Tramèr, Percy Liang, Tatsunori HashimotoICLR 2022 · 502 citations
  5. Sheared LLaMA: Accelerating Language Model Pre-training via Structured PruningMengzhou Xia, Tianyu Gao, Zhiyuan Zeng, Danqi ChenICLR 2024 · 453 citations

Related papers

  1. Synthetic Text Generation with Differential Privacy: A Simple and Practical RecipeXiang Yue, Huseyin A. Inan, Xuechen Li, Girish Kumar et al.ACL 2023 · 24 citations
  2. Differentially Private Language Models for Secure Data SharingJustus Mattern, Zhijing Jin, Benjamin Weggenmann, Bernhard Schölkopf et al.EMNLP 2022 · 16 citations
  3. Private prediction for large-scale synthetic text generationKareem Amin, Alex Bie, Weiwei Kong, Alexey Kurakin et al.EMNLP Findings 2024 · 5 citations
  4. Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data GenerationQian Ma, Sarah RajtmajerACL Findings 2026
  5. Data-Constrained Synthesis of Training Data for De-IdentificationThomas Vakili, Aron Henriksson, Hercules DalianisACL 2025 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines