Nurturing Capabilities: Unpacking the Gap in Human-Centered Evaluations of AI-Based Systems

Human-Computer Interaction (HCI) scholarship has studied how Artificial Intelligence (AI) can be leveraged to support care work(ers) by recognizing, reducing, and redistributing workload. Assessment of AI's impact on workers requires scrutiny and is a growing area of inquiry within human-centered evaluations of AI. We add to these conversations by unpacking the sociotechnical gap between the broader aspirations of workers from an AI-based system and the narrower existing definitions of success. We conducted a mixedmethods study and drew on Amartya Sen's Capability Approach to analyze the gap. We shed light on the social factors-on top of performance on evaluation metrics-that guided the AI model choice and determined whose wellbeing must be evaluated while conducting such evaluations. We argue for assessing broader achievements enabled through AI's use when conducting human-centered evaluations of AI. We discuss and recommend the dimensions to consider while conducting such evaluations.

• Human-centered computing → Empirical studies in HCI; HCI design and evaluation methods.