Lune

ICML2026顶会

Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning

Minh-Tung Luu, Hwanhee Kim, Younghwan Lee, Chang D. Yoo

摘要

Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering. Preference-based RL (PbRL) offers a promising alternative by learning reward functions from human feedback, but its scalability is hindered by high labeling costs. Inspired by advances in Video Foundation Models (ViFMs), we present Video-based Optimal Transport Preference (VOTP), a semi-supervised framework that learns effective reward functions from only a handful of labels. By leveraging optimal transport to align visual trajectories within the rich representation space of ViFMs, VOTP effectively generates high-fidelity pseudo-labels for large amounts of unlabeled data, substantially reducing human supervision. Extensive experiments across locomotion and manipulation benchmarks demonstrate the superiority of VOTP, which outperforms state-of-the-art offline PbRL methods under limited feedback budgets. We also showcase the robustness of VOTP in the presence of visual distractors and validate its utility on real robotic tasks, where it learns meaningful rewards with minimal human input. The code is available at: https: //github.com/tunglm2203/votp .

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • Offline Reinforcement Learning with Implicit Q-Learning 引用了 Decision Transformer: Reinforcement Learning via Sequence Modeling
  • Learning Transferable Visual Models From Natural Language Supervision 引用了 HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper39

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh 等ICML 2021 · 47,906 次引用
  2. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 次引用
  3. Decision Transformer: Reinforcement Learning via Sequence ModelingLili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee 等NeurIPS 2021 · 2,557 次引用
  4. HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video ClipsAntoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi 等ICCV 2019 · 1,437 次引用
  5. Offline Reinforcement Learning with Implicit Q-LearningIlya Kostrikov, Ashvin Nair, Sergey LevineICLR 2022 · 1,402 次引用

相关 Paper

  1. VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-TrainingYecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani 等ICLR 2023 · 35 次引用
  2. Optimal Transport for Offline Imitation LearningYicheng Luo, Zhengyao Jiang, Samuel Cohen, Edward Grefenstette 等ICLR 2023 · 2 次引用
  3. Robot Policy Learning with Temporal Optimal Transport RewardYuwei Fu, Haichao Zhang, Di Wu, Wei Xu 等NeurIPS 2024 · 13 次引用
  4. RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model FeedbackYufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian 等ICML 2024 · 135 次引用
  5. Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language ModelsTung Minh Luu, Younghwan Lee, Donghoon Lee, Sunho Kim 等ICML 2025
黄昏的海面,两侧是细线勾勒的悬崖