Lune

ICML2024Top-tier venue

Think Before You Act: Decision Transformers with Working Memory

Jikun Kang, Romain Laroche, Xingdi Yuan, Adam Trischler, Xue Liu, Jie Fu

Abstract

Decision Transformer-based decision-making agents have shown the ability to generalize across multiple tasks. However, their performance relies on massive data and computation. We argue that this inefficiency stems from the forgetting phenomenon, in which a model memorizes its behaviors in parameters throughout training. As a result, training on a new task may deteriorate the model's performance on previous tasks. In contrast to LLMs' implicit memory mechanism, the human brain utilizes distributed memory storage, which helps manage and organize multiple skills efficiently, mitigating the forgetting phenomenon. Inspired by this, we propose a working memory module to store, blend, and retrieve information for different downstream tasks. Evaluation results show that the proposed method improves training efficiency and generalization in Atari games and Meta-World object manipulation tasks. Moreover, we demonstrate that memory fine-tuning further enhances the adaptability of the proposed architecture. 1 Recently, with the tremendous success of decoder-only Transformer models (Brown et al., 2020; OpenAI, 2023; Dosovitskiy et al., 2021; Touvron et al., 2023) , an increasing number of researchers have focused on decoder-only Transformer-based decision-making agents. As shown with GPT-3 (Brown et al., 2020) and follow-up work (Kaplan et al., 2020; Clark et al., 2022) , the generalization of these LLMs depends significantly on the model size, i.e. the num-

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Improving Language Models by Retrieving from Trillions of Tokens cites Language Models are Few-Shot Learners
  • Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation cites Decision Transformer: Reinforcement Learning via Sequence Modeling
  • Offline Reinforcement Learning as One Big Sequence Modeling Problem cites Decision Transformer: Reinforcement Learning via Sequence Modeling
  • Offline Reinforcement Learning as One Big Sequence Modeling Problem cites Language Models are Few-Shot Learners
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
  • Decision Transformer: Reinforcement Learning via Sequence Modeling cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Decision Transformer: Reinforcement Learning via Sequence Modeling cites Language Models are Few-Shot Learners
  • DRIVE: Distributional and Retrieval-Augmented Bidding with Value Evaluation cites Decision Transformer: Reinforcement Learning via Sequence Modeling
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and EvaluationEgor Cherepanov, Nikita Kachaev, Artem Zholus, Alexey K. Kovalev et al.ICLR 2026 · 4 citations
  2. NeSyPr: Neurosymbolic Proceduralization For Efficient Embodied ReasoningWonje Choi, Jooyoung Kim, Honguk WooNeurIPS 2025 · 4 citations
  3. DRIVE: Distributional and Retrieval-Augmented Bidding with Value EvaluationMiduo Cui, Haochen Wang, Shangqin Mao, Xun Yang et al.ICML 2026 · 1 citation
  4. Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied AgentsSaehun Chun, Wonje Choi, Sera Choi, Sanghyun Ahn et al.ICML 2026

Builds on18

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. Decision Transformer: Reinforcement Learning via Sequence ModelingLili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee et al.NeurIPS 2021 · 2,557 citations
  4. Improving Language Models by Retrieving from Trillions of TokensSebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai et al.ICML 2022 · 1,629 citations
  5. Offline Reinforcement Learning as One Big Sequence Modeling ProblemMichael Janner, Qiyang Li, Sergey LevineNeurIPS 2021 · 950 citations

Related papers

  1. Learning How to Remember: A Meta-Cognitive Management Method for Structured and Transferable Agent MemorySirui Liang, Pengfei Cao, Jian Zhao, Wenhao Teng et al.ACL Findings 2026
  2. Memo: Training Memory-Efficient Embodied Agents with Reinforcement LearningGunshi Gupta, Karmesh Yadav, Zsolt Kira, Yarin Gal et al.NeurIPS 2025 · 9 citations
  3. Working Memory GraphsRicky Loynd, Roland Fernandez, Asli Celikyilmaz, Adith Swaminathan et al.ICML 2020 · 41 citations
  4. HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language ModelMengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu et al.ACL 2025
  5. Transformers are Meta-Reinforcement LearnersLuckeciano C. MeloICML 2022 · 66 citations
Dusk over the sea between two cliffs drawn in fine vertical lines