Lune

ACL Findings2023

Abstract then Play: A Skill-centric Reinforcement Learning Framework for Text-based Games

Anjie Zhu, Peng-Fei Zhang, Yi Zhang, Zi Huang, Jie Shao

Abstract

Text-based games present an exciting test-bed for reinforcement learning algorithms in the natural language environment. In these adventure games, an agent must learn to interact with the environment through text in order to accomplish tasks, facing large and combinational action space as well as partial observability issues. However, existing solutions fail to decompose the task and abstract the action autonomously, which either pre-specify the subtasks or pretrain on the human gameplay dataset. In this work, we introduce a novel skill-centric reinforcement learning framework, which is capable of abstracting the action in an end-to-end manner. To learn a more disentangled skill, we focus on the informativeness and distinguishability of the skill in accordance with the information bottleneck principle. Specifically, we introduce a discriminator to enable the skill to reflect the trajectory and push their representations onto the unit hypersphere to distribute uniformly. Moreover, a self-predictive mechanism is employed to learn inverse and forward dynamics, and a self-recovery mechanism is leveraged to refine the action representation, thus resulting in a more comprehensive perception of dynamics and more effective representations of textual state and action. Empirical experiments are carried out on the Jericho environment and the results validate the superiority against state-of-the-art baselines.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Language Model Adaption for Reinforcement Learning with Natural Language Action Space cites Graph Constrained Reinforcement Learning for Natural Language Action Spaces
  • Language Model Adaption for Reinforcement Learning with Natural Language Action Space cites Learning Dynamic Belief Graphs to Generalize on Text-Based Games
  • Language Model Adaption for Reinforcement Learning with Natural Language Action Space cites Keep CALM and Explore: Language Models for Action Generation in Text-based Games
  • Graph Constrained Reinforcement Learning for Natural Language Action Spaces cites Interactive Fiction Games: A Colossal Adventure
  • Learning Dynamic Belief Graphs to Generalize on Text-Based Games cites Interactive Fiction Games: A Colossal Adventure
  • Learning Dynamic Belief Graphs to Generalize on Text-Based Games cites Graph Constrained Reinforcement Learning for Natural Language Action Spaces
  • Keep CALM and Explore: Language Models for Action Generation in Text-based Games cites Interactive Fiction Games: A Colossal Adventure
  • Keep CALM and Explore: Language Models for Action Generation in Text-based Games cites Graph Constrained Reinforcement Learning for Natural Language Action Spaces
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Language Model Adaption for Reinforcement Learning with Natural Language Action SpaceJiangxing Wang, Jiachen Li, Xiao Han, Deheng Ye et al.ACL 2024

Builds on11

  1. Interactive Fiction Games: A Colossal AdventureMatthew J. Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, Xingdi YuanAAAI 2020 · 242 citations
  2. Graph Constrained Reinforcement Learning for Natural Language Action SpacesPrithviraj Ammanabrolu, Matthew J. HausknechtICLR 2020 · 138 citations
  3. Hierarchical Reinforcement Learning by Discovering Intrinsic OptionsJesse Zhang, Haonan Yu, Wei XuICLR 2021 · 97 citations
  4. Learning Dynamic Belief Graphs to Generalize on Text-Based GamesAshutosh Adhikari, Xingdi Yuan, Marc-Alexandre Côté, Mikulas Zelinka et al.NeurIPS 2020 · 91 citations
  5. Keep CALM and Explore: Language Models for Action Generation in Text-based GamesShunyu Yao, Rohan Rao, Matthew J. Hausknecht, Karthik NarasimhanEMNLP 2020 · 67 citations

Related papers

  1. Multi-Stage Episodic Control for Strategic Exploration in Text GamesJens Tuyls, Shunyu Yao, Sham M. Kakade, Karthik NarasimhanICLR 2022 · 30 citations
  2. Algorithmic Improvements for Deep Reinforcement Learning Applied to Interactive FictionVishal Jain, William Fedus, Hugo Larochelle, Doina Precup et al.AAAI 2020 · 32 citations
  3. Dual-Scale World Memory for LLM Agents towards Hard-Exploration ProblemsMinsoo Kim, Seung-won HwangICLR 2026
  4. Unsupervised Skill Discovery with Bottleneck Option LearningJaekyeom Kim, Seohong Park, Gunhee KimICML 2021 · 39 citations
  5. Perceiving the World: Question-guided Reinforcement Learning for Text-based GamesYunqiu Xu, Meng Fang, Ling Chen, Yali Du et al.ACL 2022 · 22 citations
Dusk over the sea between two cliffs drawn in fine vertical lines