Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the US

Recent work has highlighted the culturallycontingent nature of commonsense knowledge (Shen et al., 2024) . We introduce AMAMMERE (/A:.mA:.mu:.reI/), from the Akan word meaning 'culture.' This test set of 525 multiple-choice questions is designed to evaluate the commonsense knowledge of English LLMs, relative to the cultural contexts of Ghana and the United States. To create AMAMMERE, we select a set of multiplechoice questions (MCQs) from existing commonsense datasets and rewrite them in a multistage process involving surveys of Ghanaian and U.S. participants. In three rounds of surveys, participants from both pools are solicited to (1) write correct and incorrect answer choices, ( 2 ) rate individual answer choices on a 5-point Likert scale, and (3) select the best answer choice from the newly-constructed MCQ items, in a final validation step. By engaging participants at multiple stages, our procedure ensures that participant perspectives are incorporated both in the creation and validation of test items, resulting in high levels of agreement within each pool. We evaluate several off-the-shelf English LLMs on AMAMMERE. 1 Uniformly, models prefer answers choices that align with the preferences of U.S. annotators over Ghanaian annotators. Additionally, when test items specify a cultural context (Ghana or the U.S.), models exhibit some ability to adapt, but performance is consistently better in U.S. contexts than Ghanaian. As large resources are devoted to the advancement of English LLMs, our findings underscore the need for culturally adaptable models and evaluations to meet the needs of diverse English-speaking populations around the world. Original Version Ghanaian Version Answer Choices Unspecified Version American Version Goal: How do you dry herbs when making homemade herb oil? sol1: Place freshly washed and clean herbs in a cold refrigerator sol2: Place freshly washed and clean herbs in a food dehydrator Context: This person plans to make homemade herb oil. Queson: How will you typically dry the herbs for this? Context: This person plans to make homemade herb oil. Queson: How will Kpakpo typically dry the herbs for this in Ghana? Context: This person plans to make homemade herb oil. Queson: How will Zach typically dry the herbs for this in the USA? A. Place the herbs in a basket for the sun to dry it B. Place the herbs in a food dehydrator to dry it C. Put the herbs in an electric dryer to dry it D. Dunk the herbs in some water to dry it