Assessing Cognitive Performance Using Physiological and Facial Features: Generalizing across Contexts

Sensing and machine learning advances have enabled the unobtrusive measurement of physiological responses and facial expressions so as to estimate one's cognitive performance. This often boils down to mapping the states of the cognitive processes underpinning human cognition: physiological responses (e.g., heart rate) and facial expressions (e.g., frowning) often reflect the states of our cognitive processes. However, it remains unclear whether physiological responses and facial expressions used in one particular task (e.g., gaming) can reliably assess cognitive performance in another task (e.g., coding), because complex and diverse tasks often require varying levels and combinations of cognitive processes. In this paper, we measure the cross-task reliability of physiological and facial responses. Specifically, we assess cognitive performance based on physiological responses and facial expressions for 123 participants in 4 independent studies (3 studies for out-of-sampling training and testing, and 1 study for evaluation only): (1) a Pac-Man game, (2) an adaptive-assessment task, (3) a code-debugging task, and (4) a gaze-based game. We follow an ensemble learning approach after cross-training and cross-testing with all possible combinations of the 3 first datasets. We save the 4th dataset only for testing purposes, and we showcase how to engineer generalizable features that predict cognitive performance. Our results show that the extracted features do generalize, and can reliably predict cognitive performance across a diverse set of cognitive tasks that require different combinations of problem-solving, decision-making, and learning processes for their completion.

CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing systems and tools; Empirical studies in HCI.