Lune

ACL Findings2023

Membership Inference Attacks against Language Models via Neighbourhood Comparison

Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, Taylor Berg-Kirkpatrick

Abstract

Membership Inference attacks (MIAs) aim to predict whether a data sample was present in the training data of a machine learning model or not, and are widely used for assessing the privacy risks of language models. Most existing attacks rely on the observation that models tend to assign higher probabilities to their training samples than non-training points. However, simple thresholding of the model score in isolation tends to lead to high false-positive rates as it does not account for the intrinsic complexity of a sample. Recent work has demonstrated that reference-based attacks which compare model scores to those obtained from a reference model trained on similar data can substantially improve the performance of MIAs. However, in order to train reference models, attacks of this kind make the strong and arguably unrealistic assumption that an adversary has access to samples closely resembling the original training data. Therefore, we investigate their performance in more realistic scenarios and find that they are highly fragile in relation to the data distribution used to train reference models. To investigate whether this fragility provides a layer of safety, we propose and evaluate neighbourhood attacks, which compare model scores for a given sample to scores of synthetically generated neighbour texts and therefore eliminate the need for access to the training data distribution. We show that, in addition to being competitive with reference-based attacks that have perfect knowledge about the training data distribution, our attack clearly outperforms existing reference-free attacks as well as referencebased attacks with imperfect knowledge, which demonstrates the need for a reevaluation of the threat model of adversarial attacks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

104 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Membership Inference Attacks Against Machine Learning Models cites Deep Learning with Differential Privacy
  • LLM Dataset Inference: Did you train on my dataset? cites Membership Inference Attacks Against Machine Learning Models
  • LLM Dataset Inference: Did you train on my dataset? cites DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
  • LLM Dataset Inference: Did you train on my dataset? cites Detecting Pretraining Data from Large Language Models
  • Label-Only Membership Inference Attacks cites Membership Inference Attacks Against Machine Learning Models
  • Label-Only Membership Inference Attacks cites Deep Learning with Differential Privacy
  • Proving Test Set Contamination in Black-Box Language Models cites Membership Inference Attacks Against Machine Learning Models
  • Detecting Pretraining Data from Large Language Models cites Membership Inference Attacks Against Machine Learning Models
  • Detecting Pretraining Data from Large Language Models cites DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers104

  1. Detecting Pretraining Data from Large Language ModelsWeijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang et al.ICLR 2024 · 365 citations
  2. Formalizing and Benchmarking Prompt Injection Attacks and DefensesYupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia et al.USENIX Security 2024 · 308 citations
  3. Proving Test Set Contamination in Black-Box Language ModelsYonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak et al.ICLR 2024 · 220 citations
  4. LLM Dataset Inference: Did you train on my dataset?Pratyush Maini, Hengrui Jia, Nicolas Papernot, Adam DziedzicNeurIPS 2024 · 162 citations
  5. What makes unlearning hard and what to do about itKairan Zhao, Meghdad Kurmanji, George-Octavian Barbulescu, Eleni Triantafillou et al.NeurIPS 2024 · 115 citations

Builds on17

  1. Deep Learning with Differential PrivacyMartín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan et al.CCS 2016 · 7,620 citations
  2. Membership Inference Attacks Against Machine Learning ModelsReza Shokri, Marco Stronati, Congzheng Song, Vitaly ShmatikovS&P 2017 · 5,137 citations
  3. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  4. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability CurvatureEric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning et al.ICML 2023 · 988 citations
  5. Label-Only Membership Inference AttacksChristopher A. Choquette-Choo, Florian Tramèr, Nicholas Carlini, Nicolas PapernotICML 2021 · 628 citations

Related papers

  1. Quantifying Privacy Risks of Masked Language Models Using Membership Inference AttacksFatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick et al.EMNLP 2022 · 72 citations
  2. DF-MIA: A Distribution-Free Membership Inference Attack on Fine-Tuned Large Language ModelsZhiheng Huang, Yannan Liu, Daojing He, Yu LiAAAI 2025 · 7 citations
  3. Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt CalibrationWenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu et al.NeurIPS 2024 · 28 citations
  4. Powerful Training-Free Membership Inference Against Fine-Tuned Autoregressive Language ModelsDavid Ilic, David Stanojevic, Kostadin CvejoskiACL 2026
  5. Sampling-based Pseudo-Likelihood for Membership Inference AttacksMasahiro Kaneko, Youmi Ma, Yuki Wata, Naoaki OkazakiACL Findings 2025 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines