Wish You Were Here: Mental and Physiological Effects of Remote Music Collaboration in Mixed Reality

With face-to-face music collaboration being severely limited during the recent pandemic, mixed reality technologies and their potential to provide musicians a feeling of "being there" with their musical partner can offer tremendous opportunities. In order to assess this potential, we conducted a laboratory study in which musicians made music together in real-time while simultaneously seeing their jamming partner's mixed reality point cloud via a head-mounted display and compared mental effects such as flow, affect, and copresence to an audio-only baseline. In addition, we tracked the musicians' physiological signals and evaluated their features during times of self-reported flow. For users jamming in mixed reality, we observed a significant increase in co-presence. Regardless of the condition (mixed reality or audio-only), we observed an increase in positive affect after jamming remotely. Furthermore, we identified heart rate and HF/LF as promising features for classifying the flow state musicians experienced while making music together.

• Human-centered computing → Empirical studies in HCI; Mixed / augmented reality; Collaborative interaction; • Applied computing → Sound and music computing; • Hardware → Sensor applications and deployments.