LLM Integration in Extended Reality: A Comprehensive Review of Current Trends, Challenges, and Future Perspectives

The rapid evolution of Extended Reality (XR) technologies—encompassing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR)—has paved the way for richer and more immersive user experiences. Concurrently, the emergence of Large Language Models (LLMs), such as GPT-4, has unlocked new opportunities to enhance interactions within XR environments. This paper presents the first comprehensive review addressing the underexplored synergy between XR and LLMs, examining how the integration of these technologies can augment various aspects of human awareness: spatial, situational, social, and self-awareness. By systematically analyzing 135 papers, we synthesize and categorize the research field into seven dimensions: 1) diverse application domains, 2) types of human awareness expanded, 3) interaction paradigms between users and systems, 4) effects of LLMs in XR, 5) practices for effectively integrating LLMs into XR environments, and 6) evaluation metrics. We also discuss remaining challenges and propose future research focusing on ethical awareness.