Predicting Usability and UX based on Eye Movements: Identifying Cross-Stimuli Interaction Patterns with Machine Learning

Eye-tracking and questionnaires are typically treated as separate methods for measuring usability and user experience (UX). Recent studies show that machine learning models trained solely on eye movements can predict pragmatic and hedonic quality ratings. Building on this, this study examines which gaze patterns predict usability and UX and whether models can generalize across stimuli. Five models were trained on eye movements from 121 users browsing six websites. A feature-importance analysis revealed that saccadic patterns, such as regressions and successive forward movements, are more associated with UX, whereas longer consecutive saccades are indicative of usability. When trained separately for each website, the best-performing models achieve Matthews Correlation Coefficient scores of 0.751 and 0.780, with only small negative effect sizes on holdout data. Trained across websites, holdout scores dropped to 0.196 for usability and 0.338 for UX, suggesting that cross-stimuli generalizability is limited and, at best, achievable for hedonic interaction aspects.