UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding

The emergence of Multimodal Large Language Models has driven significant advances in Graphical User Interface (GUI) agent capabilities. Nevertheless, we recognize that existing training and inference techniques for GUI agents still suffer from a dilemma for reasoning designs, ineffective reward, and visual noise. To address these issues, we introduce UI-AGILE for enhancing GUI agents at both training and inference. For training, we propose a suite of improvements to the Reinforcement Fine-Tuning (RFT) process: 1) a "Simple Thinking" strategy to balance planning with speed and grounding accuracy, 2) a continuous reward function to incentivize high-precision grounding, and 3) a cropping-based resampling strategy to mitigate the sparse reward problem and improve learning on complex tasks. For inference, we present decomposed grounding with selection to improve grounding accuracy on highresolution displays by breaking the image into smaller, manageable parts. Experiments show that UI-AGILE exhibits better grounding performance than state-of-the-art methods on benchmarks ScreenSpot-Pro and ScreenSpot-v2, while showing strong agent capabilities. For instance, using both our training and inference enhancements brings 23% grounding accuracy improvement over the best baseline on ScreenSpot-Pro. The implementation of UI-AGILE is provided at https://github.com/KDEGroup/UI- AGILE.