SpeechCap: Leveraging Playful Impact Captions to Facilitate Interpersonal Communication in Social Virtual Reality

Fig. 1. SpeechCap showcases a novel approach to mediate voice conversations in social VR by converting speech into playful impact captions. (A) From real-time speech, SpeechCap generates impact captions with customized visual design that conveys both verbal and non-verbal information. The caption "happy" is in a warm and bright color for the positive emotions, and a "smiling" emoji further highlight the pleasant and relaxing atmosphere. (B) Users can play with impact captions like real-world objects, e.g., throwing a "cake". (C) Once the flying "cake" crashes on another user, it will explode like a firework.

Social Virtual Reality (VR) emerges as a promising platform which affords immersive, interactive, and engaging mechanisms for collaborative activities in virtual spaces. However, interpersonal communication in social VR is still limited with existing mediums and channels. To bridge the gap, we propose a novel method for mediating real-time conversations in social VR, which leverages impact captions, a type of typographic visual effect widely used in videos, to encode both verbal and non-verbal information. We first investigated the design space of impact captions by content analysis and a co-design session with four experts. We then implemented SpeechCap, a proof-of-concept system with which users can communicate with each other using speech-driven impact captions in VR. Through a user study (N=14), we evaluated the effectiveness of the visual and interaction design of speech-driven impact captions, highlighting their strengths in the interactivity and integrating verbal and non-verbal information in communication mediums. Finally, we discussed our main findings regarding visual rhetoric, interactivity, and ambiguity, and further provided design implications for facilitating interpersonal communication in social VR.