EMNLP Findings2023
ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination
Dongfang Li, Jindi Yu, Baotian Hu, Zhenran Xu, Min Zhang
Abstract
In the field of Large Language Models (LLMs), researchers are increasingly exploring their effectiveness across a wide range of tasks. However, a critical area that requires further investigation is the interpretability of these models, particularly the ability to generate rational explanations for their decisions. Most existing explanation datasets are limited to the English language and the general domain, which leads to a scarcity of linguistic diversity and a lack of resources in specialized domains, such as medical. To mitigate this, we propose ExplainCPE, a challenging medical dataset consisting of over 7K problems from Chinese Pharmacist Examination, specifically tailored to assess the model-generated explanations. From the overall results, only GPT-4 passes the pharmacist examination with a 75.7% accuracy, while other models like ChatGPT fail. Further detailed analysis of LLM-generated explanations reveals the limitations of LLMs in understanding medical text and executing computational reasoning. With the increasing importance of AI safety and trustworthiness, ExplainCPE takes a step towards improving and evaluating the interpretability of LLMs in the medical domain. The dataset is available at https: //github.com/HITsz-TMG/ExplainCPE.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting cites Training language models to follow instructions with human feedback
- Towards Self-cognitive Exploration: Metacognitive Knowledge Graph Retrieval Augmented Generation cites Language Models are Few-Shot Learners
- Towards Self-cognitive Exploration: Metacognitive Knowledge Graph Retrieval Augmented Generation cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Cited by top-tier papers2
Builds on11
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought PromptingMiles Turpin, Julian Michael, Ethan Perez, Samuel R. BowmanNeurIPS 2023 · 1,792 citations
Related papers
- EMPEC: A Comprehensive Benchmark for Evaluating Large Language Models Across Diverse Healthcare ProfessionsZheheng Luo, Chenhan Yuan, Qianqian Xie, Sophia AnaniadouACL Findings 2025 · 2 citations
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language ModelsYan Cai, Linlin Wang, Ye Wang, Gerard de Melo et al.AAAI 2024 · 42 citations
- CMedCalc-Bench: A Fine-Grained Benchmark for Chinese Medical Calculations in LLMYunyan Zhang, Zhihong Zhu, Xian WuEMNLP 2025 · 1 citation
- Towards Interpretable Mental Health Analysis with Large Language ModelsKailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie et al.EMNLP 2023 · 114 citations
- MentalGLM Series: Explainable Large Language Models for Mental Health Analysis on Chinese Social MediaWei Zhai, Nan Bai, Qing Zhao, Jianqiang Li et al.EMNLP 2025
