Harmonizing the Senses: Designing a Cross-Modal Interactive Art System to Enhance Older Adults' Affective Experiences

Multisensory stimulation promises in improving older adults’ affective experiences, yet its effectiveness depends on seamless affective congruency across sensory cues. This study investigated how visual, auditory, and kinetics correspondence and congruency shape affective experiences through two experiments. Experiment I examined timbre–color associations, showing that affective alignment strengthens perceived correspondence. Experiment II explored auditory–kinetics synchrony in a cross-modal art system, revealing no significant differences across conditions but indicating that older adults with lower cognitive abilities reported higher pleasure than higher-ability peers. Building on these results, an artificial intelligence (Al)-infused mode was integrated to transform strokes into real-time ink-style artworks, reducing cognitive effort, sustaining engagement. Findings demonstrate that AI enhances positive affect (pleasure, surprise, valence, and arousal) and mitigates negative affect (sadness, anger), with effects maximized by high sensory synchrony, providing compensatory support for users with lower cognitive abilities. These findings inform multisensory system design for older adults’ cognitive and affective needs.