Lune

ACL Findings2023

OpenPI-C: A Better Benchmark and Stronger Baseline for Open-Vocabulary State Tracking

Xueqing Wu, Sha Li, Heng Ji

Abstract

Open-vocabulary state tracking is a more practical version of state tracking that aims to track state changes of entities throughout a process without restricting the state space and entity space. OpenPI (Tandon et al., 2020) is to date the only dataset annotated for open-vocabulary state tracking. However, we identify issues with the dataset quality and evaluation metric. For the dataset, we categorize 3 types of problems on the procedure level, step level and state change level respectively, and build a clean dataset OpenPI-C using multiple rounds of human judgment. For the evaluation metric, we propose a cluster-based metric to fix the original metric's preference for repetition.

Model-wise, we enhance the seq2seq generation baseline by reinstating two key properties for state tracking: temporal dependency and entity awareness. The state of the world after an action is inherently dependent on the previous state. We model this dependency through a dynamic memory bank and allow the model to attend to the memory slots during decoding. On the other hand, the state of the world is naturally a union of the states of involved entities. Since the entities are unknown in the open-vocabulary setting, we propose a two-stage model that refines the state change prediction conditioned on entities predicted from the first stage. Empirical results show the effectiveness of our proposed model especially on the cluster-based metric.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional VideosYulei Niu, Wenliang Guo, Long Chen, Xudong Lin et al.ICLR 2024 · 26 citations

Builds on1

  1. Reasoning over Entity-Action-Location Graph for Procedural Text UnderstandingHao Huang, Xiubo Geng, Jian Pei, Guodong Long et al.ACL 2021

Related papers

  1. A Dataset for Tracking Entities in Open Domain Procedural TextNiket Tandon, Keisuke Sakaguchi, Bhavana Dalvi, Dheeraj Rajagopal et al.EMNLP 2020 · 38 citations
  2. Understand the Dynamic World: An End-to-End Knowledge Informed Framework for Open Domain Entity State TrackingMingchen Li, Lifu HuangSIGIR 2023 · 6 citations
  3. A Sequence-to-Sequence Approach to Dialogue State TrackingYue Feng, Yang Wang, Hang LiACL 2021
  4. Non-Autoregressive Dialog State TrackingHung Le, Richard Socher, Steven C. H. HoiICLR 2020 · 54 citations
  5. Efficient Dialogue State Tracking by Selectively Overwriting MemorySungdong Kim, Sohee Yang, Gyuwan Kim, Sang-Woo LeeACL 2020 · 189 citations
Dusk over the sea between two cliffs drawn in fine vertical lines