Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation

In the era of evaluating large language models (LLMs), data contamination has become an increasingly prominent concern. To address this data contamination risk, LLM benchmarking has evolved from a static to a dynamic paradigm. In this work, we conduct an indepth analysis of existing static and dynamic benchmarks for evaluating LLMs. We first examine methods that enhance static benchmarks and identify their inherent limitations. We then highlight a critical gap-the lack of standardized criteria for evaluating dynamic benchmarks. Based on this observation, we propose a series of optimal design principles for dynamic benchmarking and analyze the limitations of existing dynamic benchmarks. This survey provides a concise yet comprehensive overview of recent advancements in data contamination research, offering valuable insights and a clear guide for future research efforts. We maintain a GitHub repository to continuously collect both static and dynamic benchmarks for LLMs.