QualBench: Benchmarking Chinese LLMs with Localized Professional Qualifications for Vertical Domain Evaluation

The rapid advancement of Chinese LLMs underscores the need for vertical-domain evaluations to ensure reliable applications. However, existing benchmarks often lack domain coverage and provide limited insights into the Chinese working context. Leveraging qualification exams as a unified framework for expertise evaluation, we introduce QualBench 1 , the first multi-domain Chinese QA benchmark dedicated to localized assessment of Chinese LLMs. The dataset includes over 17,000 questions across six vertical domains, drawn from 24 Chinese qualifications to align with national policies and professional standards. Results reveal an interesting pattern of Chinese LLMs consistently surpassing non-Chinese models, with the Qwen2.5 model outperforming the more advanced GPT-4o, emphasizing the value of localized domain knowledge in meeting qualification requirements. The average accuracy of 53.98% reveals the current gaps in domain coverage within model capabilities. Furthermore, we identify performance degradation caused by LLM crowdsourcing, assess data contamination, and illustrate the effectiveness of prompt engineering and model fine-tuning, suggesting opportunities for future improvements through multi-domain RAG and Federated Learning. * Corresponding Author 1 Data and code are publicly available at https://github. com/mengze-hong/QualBench General 货币的首要或基本功能是什么？ What is the primary function of money? A. 交易工具 (Medium of Exchange) B. 储存方式 (Store of Value) C. 付款方式 (Means of Payment) D. 价值衡量标准 (Measure of Value) Localized 人民币是指____依法发行的货币。 The renminbi refers to currency issued by____ according to law. A. 中国人民银行 (People's Bank of China) B. 中国银行 (Bank of China) C. 中国工商银行 (Industrial and Commercial Bank of China) D. 银行业监督管理委员会 (China Banking Regulatory Commission)