Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination

While Large Language Models (LLMs) have increasingly assisted in historical tasks such as text processing, their capacity for professionallevel historical reasoning remains underexplored. Existing benchmarks primarily assess basic knowledge breadth or lexical understanding, failing to capture the higher-order skills-such as evidentiary reasoning-that are central to historical research. To fill this gap, we introduce ProHist-Bench, a novel benchmark anchored in the Chinese Imperial Examination (Keju) system-a comprehensive microcosm of East Asian political, social, and intellectual history spanning over 1,300 years. Developed through deep interdisciplinary collaboration, ProHist-Bench features 400 challenging, expert-curated questions across eight dynasties, accompanied by 10,891 fine-grained evaluation rubrics. Through a rigorous evaluation of 18 LLMs, we reveal a significant proficiency gap: even state-of-the-art LLMs struggle with complex historical research questions. We hope ProHist-Bench will facilitate the development of domain-specific reasoning LLMs, advance computational historical research, and further uncover the untapped potential of LLMs. We release ProHist-Bench at https://github.com/inclusionAI/ABenc h/tree/main/ProHist-Bench . * Equal contribution † Corresponding authors ……六等黜陟法是中国历史上太平天国时期制定的⼀种官吏考课(考核与升降)制度。该制度 出⾃太平天国的纲领性⽂件《天朝⽥亩制度》。太平天国为了巩固政权……The Liudeng Chuzhi Fa was an official evaluation (assessment and advancement/demotion) policy formulated during the Taiping Heavenly Kingdom period in Chinese history. This system originated from the Taiping Heavenly Kingdom's programmatic document, The Land System of the Heavenly Dynasty (Tianchao Tianmu Zhidu). The Taiping Heavenly Kingdom…… ]