ACL Findings2024
Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, Ge Li
Abstract
Recent statements about the impressive capabilities of large language models (LLMs) are usually supported by evaluating on open-access benchmarks. Considering the vast size and wide-ranging sources of LLMs' training data, it could explicitly or implicitly include test data, leading to LLMs being more susceptible to data contamination. However, due to the opacity of training data, the black-box access of models, and the rapid growth of synthetic training data, detecting and mitigating data contamination for LLMs faces significant challenges. In this paper, we propose CDD, which stands for Contamination Detection via output Distribution for LLMs. CDD necessitates only the sampled texts to detect data contamination, by identifying the peakedness of LLM's output distribution. To mitigate the impact of data contamination in evaluation, we also present TED: Trustworthy Evaluation via output Distribution, based on the correction of LLM's output distribution. To facilitate this study, we introduce two benchmarks, i.e., DET-CON and COMIEVAL, for data contamination detection and contamination mitigation evaluation tasks. Extensive experimental results show that CDD achieves the average relative improvements of 21.8%-30.2% over other contamination detection approaches in terms of Accuracy, F1 Score, and AUC metrics, and can effectively detect implicit contamination. TED substantially mitigates performance improvements up to 66.9% attributed to data contamination across various contamination setups. In realworld applications, we reveal that ChatGPT exhibits a high potential to suffer from data contamination on HumanEval benchmark. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
62 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
- CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis cites Language Models are Few-Shot Learners
- DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph cites LoRA: Low-Rank Adaptation of Large Language Models
- DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph cites Language Models are Few-Shot Learners
- DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph cites Detecting Pretraining Data from Large Language Models
- Detecting Pretraining Data from Large Language Models cites Language Models are Few-Shot Learners
Cited by top-tier papers62
- Preference Leakage: A Contamination Problem in LLM-as-a-judgeDawei Li, Renliang Sun, Yue Huang, Ming Zhong et al.ICLR 2026 · 150 citations
- lmgame-Bench: How Good are LLMs at Playing Games?Lanxiang Hu, Mingjia Huo, Yuxuan Zhang, Haoyang Yu et al.ICLR 2026 · 47 citations
- DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning GraphZhehao Zhang, Jiaao Chen, Diyi YangNeurIPS 2024 · 42 citations
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video UnderstandingZongxia Li, Xiyang Wu, Guangyao Shi, Yubin Qin et al.NeurIPS 2025 · 38 citations
- FAN: Fourier Analysis NetworksYihong Dong, Ge Li, Yongding Tao, Xue Jiang et al.NeurIPS 2025 · 37 citations
Builds on11
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Detecting Pretraining Data from Large Language ModelsWeijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang et al.ICLR 2024 · 365 citations
- Privacy Risks of General-Purpose Language ModelsXudong Pan, Mi Zhang, Shouling Ji, Min YangS&P 2020 · 291 citations
- CodeGen: An Open Large Language Model for Code with Multi-Turn Program SynthesisErik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu et al.ICLR 2023 · 234 citations
Related papers
- Controllable Contamination Detection for Reliable LLM Evaluation with Statistical GuaranteesZheng Zhang, Qi Liu, Siyuan Liang, Ning Li et al.ACL 2026
- An Open-Source Data Contamination Report for Large Language ModelsYucheng Li, Yunhao Guo, Frank Guerin, Chenghua LinEMNLP Findings 2024 · 19 citations
- Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model EvaluationQin Zhu, Qinyuan Cheng, Runyu Peng, Xiaonan Li et al.EMNLP Findings 2024 · 3 citations
- NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each BenchmarkOscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz et al.EMNLP Findings 2023 · 85 citations
- ConStat: Performance-Based Contamination Detection in Large Language ModelsJasper Dekoninck, Mark Niklas Müller, Martin T. VechevNeurIPS 2024 · 40 citations
