ETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation
Zhexin Zhang, Jiaxin Wen, Minlie Huang
Abstract
Large pre-trained language models achieve impressive results across many tasks. However, recent works point out that pre-trained language models may memorize a considerable fraction of their training data, leading to the privacy risk of information leakage. In this paper, we propose a method named ETHICIST for targeted training data Extraction THrough loss smoothed soft prompting and calIbrated ConfIdence eSTimation, investigating how to recover the suffix in the training data when given a prefix. To elicit memorization in the attacked model, we tune soft prompt embeddings while keeping the model fixed. We further propose a smoothing loss that smooths the loss distribution of the suffix tokens to make it easier to sample the correct suffix. In order to select the most probable suffix from a collection of sampled suffixes and estimate the prediction confidence, we propose a calibrated confidence estimation method, which normalizes the confidence of the generated suffixes with a local estimation. We show that ETHICIST significantly improves the extraction performance on a recently proposed public benchmark. We also investigate several factors influencing the data extraction performance, including decoding strategy, model scale, prefix length, and suffix length. Our code is available at https://github.com/ thu-coai/Targeted-Data-Extraction .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
12 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Membership Inference Attacks Against Machine Learning Models cites Deep Learning with Differential Privacy
- Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization cites Extracting Training Data from Large Language Models
- Unveiling the Implicit Toxicity in Large Language Models cites Language Models are Few-Shot Learners
- Language Models are Few-Shot Learners cites The Curious Case of Neural Text Degeneration
- Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLM cites Deep Learning with Differential Privacy
- Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLM cites Extracting Training Data from Large Language Models
- The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Risks cites Language Models are Few-Shot Learners
- The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Risks cites Deep Learning with Differential Privacy
- The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Risks cites Extracting Training Data from Large Language Models
- Extracting Training Data from Large Language Models cites Membership Inference Attacks Against Machine Learning Models
- Extracting Training Data from Large Language Models cites Language Models are Few-Shot Learners
- Extracting Training Data from Large Language Models cites Deep Learning with Differential Privacy
- Extracting Training Data from Large Language Models cites The Curious Case of Neural Text Degeneration
Cited by top-tier papers12
- Unveiling the Implicit Toxicity in Large Language ModelsJiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang et al.EMNLP 2023 · 21 citations
- How Should We Enhance the Safety of Large Reasoning Models: An Empirical StudyZhexin Zhang, Xian Qi Loye, Victor Shea-Jay Huang, Junxiao Yang et al.ACL 2026 · 20 citations
- The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy RisksXiaoyi Chen, Siyuan Tang, Rui Zhu, Shijun Yan et al.CCS 2024 · 18 citations
- Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy OptimizationXiyue Peng, Hengquan Guo, Jiawei Zhang, Dongqing Zou et al.NeurIPS 2025 · 9 citations
- Unlearned but Not Forgotten: Data Extraction after Exact Unlearning in LLMXiaoyu Wu, Yifei Pang, Terrance Liu, Steven Z. WuNeurIPS 2025 · 8 citations
Builds on15
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Deep Learning with Differential PrivacyMartín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan et al.CCS 2016 · 7,620 citations
- Membership Inference Attacks Against Machine Learning ModelsReza Shokri, Marco Stronati, Congzheng Song, Vitaly ShmatikovS&P 2017 · 5,137 citations
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- Extracting Training Data from Large Language ModelsNicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski et al.USENIX Security 2021 · 2,866 citations
Related papers
- Private Investigator: Extracting Personally Identifiable Information from Large Language Models Using Optimized PromptsSeongho Keum, Dongwon Shin, Leo Marchyok, Sanghyun Hong et al.USENIX Security 2025
- Unlocking Memorization in Large Language Models with Dynamic Soft PromptingZhepeng Wang, Runxue Bao, Yawen Wu, Jackson Taylor et al.EMNLP 2024 · 1 citation
- Retracing the Past: LLMs Emit Training Data When They Get LostMyeongseob Ko, Nikhil Reddy Billa, Adam Nguyen, Charles Fleming et al.EMNLP 2025 · 1 citation
- Teach LLMs to Phish: Stealing Private Information from Language ModelsAshwinee Panda, Christopher A. Choquette-Choo, Zhengming Zhang, Yaoqing Yang et al.ICLR 2024 · 41 citations
- Fragments to Facts: Partial-Information Fragment Inference from LLMsLucas Rosenblatt, Bin Han, Robert Wolfe, Bill HoweICML 2025
