Lune

SIGIR2023Top-tier venue

Understand the Dynamic World: An End-to-End Knowledge Informed Framework for Open Domain Entity State Tracking

Mingchen Li, Lifu Huang

Abstract

Open domain entity state tracking aims to predict reasonable state changes of entities (i.e., [attribute] of [entity] was [before_state] and [after_state] afterwards) given the action descriptions. It's important to many reasoning tasks to support human everyday activities. However, it's challenging as the model needs to predict an arbitrary number of entity state changes caused by the action while most of the entities are implicitly relevant to the actions and their attributes as well as states are from open vocabularies. To tackle these challenges, we propose a novel end-to-end Knowledge Informed framework for open domain Entity State Tracking, namely Kiest, which explicitly retrieves the relevant entities and attributes from external knowledge graph (i.e., ConceptNet) and incorporates them to autoregressively generate all the entity state changes with a novel dynamic knowledge grained encoder-decoder framework. To enforce the logical coherence among the predicted entities, attributes, and states, we design a new constraint decoding strategy and employ a coherence reward to improve the decoding process. Experimental results show that our proposed Kiest framework significantly outperforms the strong baselines on the public benchmark dataset -OpenPI. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional VideosYulei Niu, Wenliang Guo, Long Chen, Xudong Lin et al.ICLR 2024 · 26 citations

Builds on13

  1. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning FrameworkPeng Wang, An Yang, Rui Men, Junyang Lin et al.ICML 2022 · 1,058 citations
  2. K-BERT: Enabling Language Representation with Knowledge GraphWeijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang et al.AAAI 2020 · 898 citations
  3. Why Gradient Clipping Accelerates Training: A Theoretical Justification for AdaptivityJingzhao Zhang, Tianxing He, Suvrit Sra, Ali JadbabaieICLR 2020 · 598 citations
  4. Minimize Exposure Bias of Seq2Seq Models in Joint Entity and Relation ExtractionRanran Haoran Zhang, Qianying Liu, Aysa Xuemo Fan, Heng Ji et al.EMNLP Findings 2020 · 52 citations
  5. Enhanced Story Comprehension for Large Language Models through Dynamic Document-Based Knowledge GraphsBerkeley R. Andrus, Yeganeh Nasiri, Shilong Cui, Benjamin Cullen et al.AAAI 2022 · 46 citations

Related papers

  1. OpenPI-C: A Better Benchmark and Stronger Baseline for Open-Vocabulary State TrackingXueqing Wu, Sha Li, Heng JiACL Findings 2023 · 2 citations
  2. A Dataset for Tracking Entities in Open Domain Procedural TextNiket Tandon, Keisuke Sakaguchi, Bhavana Dalvi, Dheeraj Rajagopal et al.EMNLP 2020 · 38 citations
  3. Knowledge-guided Open Attribute Value Extraction with Reinforcement LearningYe Liu, Sheng Zhang, Rui Song, Suo Feng et al.EMNLP 2020 · 4 citations
  4. MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning ModelsVanya Cohen, Ray MooneyICML 2026 · 2 citations
  5. Open-Domain Contextual Link Prediction and its Complementarity with Entailment GraphsMohammad Javad Hosseini, Shay B. Cohen, Mark Johnson, Mark SteedmanEMNLP Findings 2021 · 8 citations
Dusk over the sea between two cliffs drawn in fine vertical lines