Mixture of Inputs: Text Generation Beyond Discrete Token Sampling

In standard autoregressive generation, an LLM predicts the next-token distribution, samples a discrete token, and then discards the distribution, passing only the sampled token as new input. To preserve this distribution's rich information, we propose Mixture of Inputs (MOI), a training-free method for autoregressive generation. After generating a token following the standard paradigm, we construct a new input that blends the generated discrete token with the previously discarded token distribution. Specifically, we employ a Bayesian estimation method that treats the token distribution as the prior, the sampled token as the observation, and replaces the conventional one-hot vector with the continuous posterior expectation as the new model input. MOI allows the model to maintain a richer internal representation throughout the generation process, resulting in improved text quality and reasoning capabilities. On mathematical reasoning, code generation, and PhDlevel QA tasks, MOI consistently improves performance across multiple models including QwQ-32B, Nemotron-Super-49B, Gemma-3-27B, and DAPO-Qwen-32B, with no additional training and negligible computational overhead.

On the other hand, human thinking first occurs in a high-dimensional and fluid manner before being articulated as natural language. Inspired by this cognitive process, we explore methods to enable LLMs to utilize not only articulated natural language but also partially-formed ideas, competing possibilities, and conceptual associations that exist in a probabilistic space before crystallizing into words.

Specifically, we propose Mixture of Inputs (MOI), a novel approach that takes as input not only a discrete, sampled token but also the sampled token's distribution. This preserves the model's uncertainty and allows it to conduct inner speech in a high-dimensional space. We employ a Bayesian estimation method, treating the token distribution as the prior and the sampled token as the observation, then replacing the conventional one-hot vector with the continuous posterior expectation. With this posterior expectation, a weighted average embedding is passed as the new input to subsequent prediction steps.