Comparing Referential Interactions with an Intelligent Assistant in Virtual Reality

Intelligent assistants (IAs) in virtual reality (VR) environments offer novel ways of interaction between humans and artificial intelligence (AI). Human-like conversational abilities change the way we interact with AI. In human communication, multimodal referential interaction, such as using eye gaze or pointing with deictic expressions, or by using descriptive speech, is essential for achieving a shared understanding of the space surrounding the interaction partners. By facilitating efficient and intuitive referential interaction among users and IAs, user engagement and overall effectiveness could be enhanced.This paper compares three referential interaction methods in VR to determine user preferences: (i) verbal communication, (ii) gaze, and (iii) pointing. We conducted a user study with 39 participants in VR, where information about scene objects had to be gathered using these different referential interaction methods. The results reveal that gaze performed well in terms of performance (both time and number of prompts) and usability measures (as assessed by the UEQ and NASA-TLX). While verbal interaction was less physically straining than pointing, it required greater mental effort. Although there were no significant differences in the overall preferences, a qualitative analysis indicated differences in individual inclination. Enriching the referential methods with feedback and enabling multimodal interactions are promising methods to improve usability and decrease task load further.