Lune

UIST2023Top-tier venue

Pantœnna: Mouth pose estimation for ar/vr headsets using low-profile antenna and impedance characteristic sensing

Daehwa Kim, Chris Harrison

Abstract

Methods for faithfully capturing a user’s holistic pose have immediate uses in AR/VR, ranging from multimodal input to expressive avatars. Although body-tracking has received the most attention, the mouth is also of particular importance, given that it is the channel for both speech and facial expression. In this work, we describe a new RF-based approach for capturing mouth pose using an antenna integrated into the underside of a VR/AR headset. Our approach side-steps privacy issues inherent in camera-based methods, while simultaneously supporting silent facial expressions that audio-based methods cannot. Further, compared to bio-sensing methods such as EMG and EIT, our method requires no contact with the wearer’s body and can be fully self-contained in the headset, offering a high degree of physical robustness and user practicality. We detail our implementation along with results from two user studies, which show a mean 3D error of 2.6 mm for 11 mouth keypoints across worn sessions without re-calibration.

Ask Lune

Know where this paper stands in minutes, not an afternoon.

Lune can read the top-tier papers that cite it and the ones closest to it, and tells you what it claims, what came after and where it falls short, each point cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

its abstract · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • OFERA: Blendshape-Driven 3D Gaussian Control for Occluded Facial Expression to Realistic Avatars in VR cites EyeEcho: Continuous and Low-power Facial Expression Tracking on Glasses
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. EyeEcho: Continuous and Low-power Facial Expression Tracking on GlassesKe Li, Ruidong Zhang, Siyuan Chen, Boao Chen et al.CHI 2024 · 27 citations
  2. OFERA: Blendshape-Driven 3D Gaussian Control for Occluded Facial Expression to Realistic Avatars in VRSeokhwan Yang, Boram Yoon, Seoyoung Kang, Hail Song et al.IEEE VR 2026 · 1 citation

Related papers

  1. EtherPose: Continuous Hand Pose Tracking with Wrist-Worn Antenna Impedance Characteristic SensingDaehwa Kim, Chris HarrisonUIST 2022 · 35 citations
  2. Mouth Haptics in VR using a Headset Ultrasound Phased ArrayVivian Shen, Craig D. Shultz, Chris HarrisonCHI 2022 · 62 citations
  3. MI-Poser: Human Body Pose Tracking Using Magnetic and Inertial Sensor Fusion with Metal Interference MitigationRiku Arakawa, Bing Zhou, Gurunandan Krishnan, Mayank Goel et al.UbiComp 2023 · 15 citations
  4. C-Face: Continuously Reconstructing Facial Expressions by Deep Learning Contours of the Face with Ear-mounted Miniature CamerasTuochao Chen, Benjamin Steeper, Kinan Alsheikh, Songyun Tao et al.UIST 2020 · 49 citations
  5. EM-POSE: 3D Human Pose Estimation from Sparse Electromagnetic TrackersManuel Kaufmann, Yi Zhao, Chengcheng Tang, Lingling Tao et al.ICCV 2021 · 48 citations
Dusk over the sea between two cliffs drawn in fine vertical lines