AIPO: Adaptive Information Guided Token-Level Reinforcement Learning for Large Language Model Reasoning

Reinforcement Learning with Verifiable Rewards (RLVR) improves the reasoning capability of Large Language Models (LLMs). Current RLVR trains LLMs on all generated tokens, rather than exploring which tokens actually contribute to reasoning. We propose AIPO (Adaptive-Information Policy Optimization), which focuses updates on those decisive tokens discovered on the fly. AIPO estimates each hidden state's mutual information to score tokens. Policy gradients are then computed only on these critical tokens, using an advantage that verifiable correctness. To improve the efficiency of mutual-information estimation, AIPO adopts a Random-Fourier approximation of the Hilbert-Schmidt Independence Criterion. Across five math and science benchmarks, AIPO yields up to +20% accuracy over strong RLVR baselines while updating merely 10% of tokens, demonstrating superior efficiency and effectiveness. Our findings highlight the importance of information-driven token selection for efficient and effective reinforcement learning of LLM reasoning.