Lune

UbiComp2026Top-tier venue

MutterMeter: Earable Sensing of Self-Talk through Hierarchical Acoustic-Linguistic Fusion toward Self-Talk Interventions

Euihyeok Lee, Seonghyeon Kim, SangHun Im, Heung-Seon Oh, Seungwoo Kang

Abstract

Self-talk is a meaningful psychological signal that reflects individuals' emotions, motivations, and cognitive states, yet remains difficult to capture in daily life. It often emerges spontaneously during moments of pressure, concentration, or repeated decision-making, making it a relevant target for behavioral and cognitive interventions such as Educational Self-Talk Intervention (ESTI). However, ESTI largely depends on expert observation or retrospective self-reports, which are limited in capturing fleeting self-talk patterns and hinder its scalability in real-world settings. To address this gap, we present MutterMeter, a mobile self-talk detection system that continuously analyzes audio from earable microphones and classifies utterances into Negative self-talk, Positive self-talk, and Others. Focusing on tennis as a high-pressure and cognitively demanding testbed, MutterMeter employs a hierarchical framework that integrates acoustic, linguistic, and contextual information while adaptively balancing accuracy and efficiency. Evaluated on a first-of-its-kind dataset we collected (34.5h, n=29), MutterMeter achieves a macro-averaged F 1 score of 0.815 across subjects, outperforming baselines such as LLM-based models and speech emotion recognition models. Moreover, its adaptive processing pipeline reduces unnecessary computation by finalizing confident cases early, while maintaining robust detection performance.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation cites Robust Speech Recognition via Large-Scale Weak Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
  2. Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction TuningZebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang et al.NeurIPS 2024 · 293 citations
  3. Disentangled Representation Learning for Multimodal Emotion RecognitionDingkang Yang, Shuai Huang, Haopeng Kuang, Yangtao Du et al.ACM MM 2022 · 260 citations
  4. Gated Fully Fusion for Semantic SegmentationXiangtai Li, Houlong Zhao, Lei Han, Yunhai Tong et al.AAAI 2020 · 229 citations
  5. emotion2vec: Self-Supervised Pre-Training for Speech Emotion RepresentationZiyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li et al.ACL Findings 2024 · 157 citations

Related papers

  1. Earinter: A Closed-Loop System for Eating Pace Regulation with Just-in-Time Intervention Using Commodity EarbudsJun Fang, Ka I. Chan, Xiyuxing Zhang, Yuntao Wang et al.UbiComp 2026
  2. EarMeter: Continuous Respiration Volume Monitoring with EarablesYang Liu, Qiang Yang, Kayla-Jade Butkow, Jake Stuchbury-Wass et al.UbiComp 2026 · 6 citations
  3. OSMO: Open-vocabulary Self-eMOtion TrackingMohamed Abdelfattah, Bugra Tekin, Fadime Sener, Necati Cihan Camgoz et al.CVPR 2026
  4. Automated Face-To-Face Conversation Detection on a Commodity Smartwatch with Acoustic SensingDawei Liang, Alice Zhang, Edison ThomazUbiComp 2023 · 11 citations
  5. UFace: Your Smartphone Can "Hear" Your Facial Expression!Shuning Wang, Linghui Zhong, Yongjian Fu, Lili Chen et al.UbiComp 2024 · 17 citations
Dusk over the sea between two cliffs drawn in fine vertical lines