ChronoBias: A Benchmark for Evaluating Temporal Group Bias in the Time-sensitive Knowledge of Large Language Models

In this paper, we propose ChronoBias, a novel benchmark for evaluating time-conditional group bias in the time-sensitive knowledge of large language models (LLMs). Our benchmark is constructed via a template-based semiautomated generation method, balancing the quality-quantity trade-off in existing benchmark curation approaches. For knowledge that changes over time, time-conditional group bias exhibits varying patterns across time intervals, evident in both the best-and worst-performing groups and in the bias metric itself. In addition to parametric knowledge bias-which influences group bias across all time intervalswe identify time-sensitivity bias as an additional factor after a model's knowledge cutoff, accounting for much of the variation in timeconditional group bias over time. Since both biases are irreducible, retrieval-augmented generation (RAG) can be a promising approach, as it can address post-cutoff knowledge and better leverage pretraining knowledge that is underrepresented in the model parameters. While RAG improves both overall performance and group bias, we observe that the disparate patterns of time-conditional group bias still persist. Therefore, through extensive experiments with various model configurations, we illustrate how accurate and fair RAG-based LLMs should behave and provide actionable guidelines toward constructing such ideal models.