ACL Findings2023
Multimedia Generative Script Learning for Task Planning
Qingyun Wang, Manling Li, Hou Pong Chan, Lifu Huang, Julia Hockenmaier, Girish Chowdhary, Heng Ji
Abstract
Goal-oriented generative script learning aims to generate subsequent steps to reach a particular goal, which is an essential task to assist robots or humans in performing stereotypical activities.An important aspect of this process is the ability to capture historical states visually, which provides detailed information that is not covered by text and will guide subsequent steps.Therefore, we propose a new task, Multimedia Generative Script Learning, to generate subsequent steps by tracking historical states in both text and vision modalities, as well as presenting the first benchmark containing 5,652 tasks and 79,089 multimedia steps.This task is challenging in three aspects: the multimedia challenge of capturing the visual states in images, the induction challenge of performing unseen tasks, and the diversity challenge of covering different information in individual steps.We propose to encode visual state changes through a selective multimedia encoder to address the multimedia challenge, transfer knowledge from previously observed tasks using a retrieval-augmented decoder to overcome the induction challenge, and further present distinct information at each step by optimizing a diversity-oriented contrastive learning objective.We define metrics to evaluate both generation and inductive quality.Experiment results demonstrate that our approach significantly outperforms strong baselines 1 . Goal: make a bubbling brook braceletStep 1: gather the necessary items. Next Step? Step History: Step 5: cut the thread.Ground Truth: string the beads and clasp onto the string in the following order: 1 medium, 1 dark, 1 end of clasp, 1 dark, and 1 medium.BART: measure the length of the bead. Our Model: thread the beads
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflection cites Learning Transferable Visual Models From Natural Language Supervision
- Non-Sequential Graph Script Induction via Multimedia Grounding cites Procedure-Aware Pretraining for Instructional Video Understanding
- Procedure-Aware Pretraining for Instructional Video Understanding cites Learning Transferable Visual Models From Natural Language Supervision
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites Learning Transferable Visual Models From Natural Language Supervision
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Cited by top-tier papers6
- EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product AssociationWeiqi Wang, Limeng Cui, Xin Liu, Sreyashi Nag et al.ACL 2025 · 15 citations
- The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT InteractionsSiru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong et al.EMNLP 2023 · 10 citations
- Non-Sequential Graph Script Induction via Multimedia GroundingYu Zhou, Sha Li, Manling Li, Xudong Lin et al.ACL 2023 · 5 citations
- Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflectionYucheng Suo, Fan Ma, Kaixin Shen, Linchao Zhu et al.ICLR 2025
- Procedure-Aware Pretraining for Instructional Video UnderstandingHonglu Zhou, Roberto Martín-Martín, Mubbasir Kapadia, Silvio Savarese et al.CVPR 2023
Builds on27
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations
- Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied AgentsWenlong Huang, Pieter Abbeel, Deepak Pathak, Igor MordatchICML 2022 · 1,539 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
Related papers
- MULTISCRIPT: Multimodal Script Learning for Supporting Open Domain Everyday TasksJingyuan Qi, Minqian Liu, Ying Shen, Zhiyang Xu et al.AAAI 2024 · 3 citations
- Take a Break in the Middle: Investigating Subgoals towards Hierarchical Script GenerationXinze Li, Yixin Cao, Muhao Chen, Aixin SunACL Findings 2023 · 1 citation
- SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional VideosYulei Niu, Wenliang Guo, Long Chen, Xudong Lin et al.ICLR 2024 · 26 citations
- Distilling Script Knowledge from Large Language Models for Constrained Language PlanningSiyu Yuan, Jiangjie Chen, Ziquan Fu, Xuyang Ge et al.ACL 2023 · 14 citations
- Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QAGuanhua Chen, Yutong Yao, Shenghe Sun, Ci-Jun Gao et al.ACL Findings 2026
