Post-Training LLMs as Better Decision-Making Agents: A Regret-Minimization Approach

Large language models (LLMs) are increasingly deployed as "agents" for decision-making (DM) in interactive and dynamic environments. Yet, since they were not originally designed for DM, recent studies have shown that LLMs can struggle even in basic online DM problems, failing to exhibit desirable behaviors such as achieving a low regret value or an effective explorationexploitation tradeoff. To address this issue, we introduce Iterative Regret-Minimization Fine-Tuning (Iterative RMFT), a new post-training procedure that repeatedly distills low-regret decision trajectories back into the base model. At each iteration, the model rolls out several decision trajectories for a given online DM task. Iterative RMFT then selects the k-lowest regret trajectories, and trains the model on those trajectories via supervised fine-tuning. Unlike prior approaches that either (a) distill action sequences from known online DM algorithms, and/or (b) rely on manually crafted chain-of-thought outputs structured around known and fixed algorithms, our approach leverages the regret metric to automatically elicit the model's DM ability, and integrates the model's self-generated reasoning rationales. This reliance on model-generated reasoning avoids rigid output format engineering, and provides more flexible training signals in the natural language format. Empirical results show that Iterative RMFT can improve LLMs' DM performance across a spectrum of models-from Transformers with numerical input/output, to lightweight open-weight LLMs, and to the more advanced closed-weight LLM, GPT-4o mini. Notably, the flexibility that Iterative RMFT offers regarding output and reasoning formats allows the trained models to naturally exhibit generalization performance across tasks varying in time horizon, action space size, reward generation processes, and DM contexts/scenarios that are described by natural language. Finally, we provide a theoretical insight into how a single-layer attention Transformer model may lead to a no-regret learner under our training paradigm, in a simplified setting. Overall, we position our approach as an initial exploration, calling for more principled and novel post-training paradigms for LLMs when it comes to addressing DM tasks.