Lune

EMNLP Findings2024

LLM-Based Offline Learning for Embodied Agents via Consistency-Guided Reward Ensemble

Yujeong Lee, Sangwoo Shin, Wei-Jin Park, Honguk Woo

Abstract

Employing large language models (LLMs) to enable embodied agents has become popular, yet it presents several limitations in practice.In this work, rather than using LLMs directly as agents, we explore their use as tools for embodied agent learning.Specifically, to train separate agents via offline reinforcement learning (RL), an LLM is used to provide dense reward feedback on individual actions in training datasets.In doing so, we present a consistencyguided reward ensemble framework (COREN), designed for tackling difficulties in grounding LLM-generated estimates to the target environment domain.The framework employs an adaptive ensemble of spatio-temporally consistent rewards to derive domain-grounded rewards in the training datasets, thus enabling effective offline learning of embodied agents in different environment domains.Experiments with the VirtualHome benchmark demonstrate that COREN significantly outperforms other offline RL agents, and it also achieves comparable performance to state-of-the-art LLM-based agents with 8B parameters, despite COREN having only 117M parameters for the agent policy network and using LLMs only for training.𝑜 !𝑜" … 𝑟 !# " 𝑟 ## " 𝑟 ! " 𝑟 # " Prompt 𝒫 !Majority vote ⋯ Trajectory 𝜏 for instruction 𝑖.𝑯𝒏 𝝉 = 𝑙

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Eureka: Human-Level Reward Design via Coding Large Language Models cites Guiding Pretraining in Reinforcement Learning with Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on18

  1. Conservative Q-Learning for Offline Reinforcement LearningAviral Kumar, Aurick Zhou, George Tucker, Sergey LevineNeurIPS 2020 · 2,881 citations
  2. LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language ModelsChan Hee Song, Brian M. Sadler, Jiaman Wu, Wei-Lun Chao et al.ICCV 2023 · 685 citations
  3. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
  4. Eureka: Human-Level Reward Design via Coding Large Language ModelsYecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang et al.ICLR 2024 · 582 citations
  5. Guiding Pretraining in Reinforcement Learning with Large Language ModelsYuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas et al.ICML 2023 · 257 citations

Related papers

  1. Code as Reward: Empowering Reinforcement Learning with VLMsDavid Venuto, Mohammad Sami Nur Islam, Martin Klissarov, Doina Precup et al.ICML 2024 · 29 citations
  2. LARM: Large Auto-Regressive Model for Long-Horizon Embodied IntelligenceZhuoling Li, Xiaogang Xu, Zhenhua Xu, Ser-Nam Lim et al.ICML 2025
  3. Master Skill Learning with Policy-Grounded Synergy of LLM-based Reward Shaping and ExploringYanbin Chang, Junfan Lin, Jie Jiang, Runhao Zeng et al.ICLR 2026
  4. Teaching Models to Improve on TapeLiat Bezalel, Eyal Orgad, Amir GlobersonAAAI 2025
  5. ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning EngineeringZexi Liu, Jingyi Chai, Xinyu Zhu, shuo tang et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines