CURE: A dataset for Clinical Understanding & Retrieval Evaluation

Given the dominance of dense retrievers which do not generalize well beyond their training dataset distributions, domain-specific test sets are essential in evaluating retrieval performance.Few test datasets are available for retrieval systems intended for use by healthcare providers in a point-of-care setting.To fill this gap we have collaborated with medical professionals to create CURE, a test dataset for passage retrieval composed of 2,000 expert written queries spanning 10 medical domains with a monolingual (English) and two cross-lingual (French/Spanish to English) conditions.In this paper, we describe how CURE was constructed and provide baseline results to showcase its effectiveness as an evaluation tool.CURE is published with a CC BY-NC 4.0 license and can be accessed on Hugging Face 1 and as a retrieval task on MTEB.