Shanks: Simultaneous Hearing and Thinking for Spoken Language Models

Current large language models (LLMs) and spoken language models (SLMs) begin thinking and taking actions only after the user has finished their turn. This disables the model from interacting with the user during the user's turn and can lead to a high response latency for waiting for the model to think. Consequently, thinking after receiving the full input is not suitable for speech-to-speech interaction, where real-time and low-latency interaction is important. We address the above issue by drawing inspiration from the fact that humans can naturally "think while listening". In this paper, we propose SHANKS, a general inference framework that enables SLMs to generate unspoken chain-of-thought reasoning when listening to the user input. SHANKS streams the input speech in fixed-duration chunks and, as soon as a chunk is received, generates unspoken reasoning based on all previous speech and reasoning, while the user continues speaking. SHANKS uses unspoken reasoning to determine whether to interrupt the user and make tool calls to complete the task. We demonstrate that SHANKS enhances the real-time user-SLM interaction in two scenarios: (1) When the user is presenting their step-by-step solution to a math problem, SHANKS can listen to and reason over the user's speech and make an interruption when the user makes a mistake. SHANKS interrupts the user 37.1% more accurately compared with a baseline that interrupts the user without thinking. (2) In a tool-augmented dialogue scenario, where the model needs to make tool calls to achieve the user's request, SHANKS can complete 56.9% of the tool calls before the user even ends their turn. Overall, SHANKS is a step toward models that keep thinking throughout the conversation, not only after a turn ends. Animated illustrations of SHANKS can be found at https: //d223302.github.io/SHANKS/ .