PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs
Artem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, Vivek Kumar
Abstract
Current multimodal large language models (LLMs) process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Conversely, existing spatial audio models are constrained to fixed microphone geometries, preventing their deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is inherently agnostic to microphone geometry. By taking raw multichannel audio and microphone coordinates as inputs, PhaseCoder performs accurate localization and produces robust spatial embeddings. We demonstrate that the Gemma 3n LLM can be fine-tuned to process and reason over the "Spatial Audio Tokens" produced by our encoder. PhaseCoder achieves state-of-the-art results on microphoneinvariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array. Our models are publicly released at https://github.com/ google-deepmind/phasecoder .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Learning Spatially-Aware Language and Audio Embeddings cites BAT: Learning to Reason about Spatial Sounds with Large Language Models
- BAT: Learning to Reason about Spatial Sounds with Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
Builds on7
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language ModelsSreyan Ghosh, Arushi Goel, Jaehyeon Kim, Sonal Kumar et al.NeurIPS 2025 · 299 citations
- BAT: Learning to Reason about Spatial Sounds with Large Language ModelsZhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen et al.ICML 2024 · 42 citations
- Learning Spatially-Aware Language and Audio EmbeddingsBhavika Devnani, Skyler Seto, Zakaria Aldeneh, Alessandro Toso et al.NeurIPS 2024 · 31 citations
Related papers
- Textual Supervision Enhances Geospatial Representations in Vision-Language ModelsMarcelo Sartori Locatelli, Fernando Tonucci, Jea Kwon, Luiz Felipe Vecchietti et al.ICML 2026
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language ModelsSubrata Biswas, Mohammad Nur Hossain Khan, Bashima IslamICLR 2026 · 9 citations
- JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical EnvironmentsZhan Liu, Changli Tang, Yuxin WANG, Zhiyuan Zhu et al.ICML 2026
- SpatialLM: Training Large Language Models for Structured Indoor ModelingYongsen Mao, Junhao Zhong, Chuan Fang, Jia Zheng et al.NeurIPS 2025 · 89 citations
- Physics-Informed Audio-Geometry-Grid Representation Learning for Universal Sound Source LocalizationMin-Sang Baek, Gyeong-Su Kim, Donghyun Kim, Joon-Hyuk ChangICLR 2026
