Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents

Large language model (LLM) agents have demonstrated strong capabilities in complex interactive decision-making tasks. However, existing LLM agents typically rely on increasingly long interaction histories, resulting in high computational cost and limited scalability. In this paper, we propose STEP-HRL, a hierarchical reinforcement learning (HRL) framework that enables step-level learning by conditioning only on single-step transitions rather than full interaction histories. STEP-HRL structures tasks hierarchically, using completed subtasks to represent global progress of overall task. By introducing a local progress module, it also iteratively and selectively summarizes interaction history within each subtask to produce a compact summary of local progress. Together, these components yield augmented step-level transitions for both high-level and low-level policies. Experimental results on ScienceWorld and ALFWorld benchmarks consistently demonstrate that STEP-HRL substantially outperforms baselines in terms of performance and generalization while reducing token usage. Our code is available at https: //github.com/TonyStark042/STEP-HRL . D Case Study Trajectory Comparison Normal Task Description: Your task is to find a(n) non-living thing. First, focus on the thing. Then, move it to the purple box in the bathroom. Observation: This room is called the kitchen. In it, you see the agent, a substance called air, a chair, a counter, a cupboard, a freezer, a fridge, a glass jar, a lighter, an oven, a painting, a sink, soap, a stopwatch, a stove, a table with a glass cup, and a thermometer. You also see: a door to the bathroom (closed), a door to the hallway (closed), and a door to the outside (closed). Action: open door to bathroom Reward: 0.08 Observation: The door is now open. Action: go to bathroom Reward: 0.17 Observation: This room is called the bathroom. In it, you see a bathtub, a drawing, a glass cup, a purple box, a sink, and a toilet. Action: focus on cup Reward: 0.50 Observation: You focus on the glass cup. Action: move cup to purple box Reward: 0.25 Observation: You move the glass cup to the purple box. done.