Lune

EMNLP Findings2020

Pragmatic Issue-Sensitive Image Captioning

Allen Nie, Reuben Cohn-Gordon, Christopher Potts

Abstract

Image captioning systems need to produce texts that are not only true but also relevant in that they are properly aligned with the current issues. For instance, in a newspaper article about a sports event, a caption that not only identifies the player in a picture but also comments on their ethnicity could create unwanted reader reactions. To address this, we propose Issue-Sensitive Image Captioning (ISIC). In ISIC, the captioner is given a target image and an issue, which is a set of images partitioned in a way that specifies what information is relevant. For the sports article, we could construct a partition that places images into equivalence classes based on player position. To model this task, we use an extension of the Rational Speech Acts model. Our extension is built on top of state-of-the-art pretrained neural image captioners and explicitly uses image partitions to control caption generation. In both automatic and human evaluations, we show that these models generate captions that are descriptive and issue-sensitive. Finally, we show how ISIC can complement and enrich the related task of Visual Question Answering.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ContextRef: Evaluating Referenceless Metrics for Image Description Generation cites Dealing with Semantic Underspecification in Multimodal NLP
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. How to talk so AI will learn: Instructions, descriptions, and autonomyTheodore R. Sumers, Robert D. Hawkins, Mark K. Ho, Tom Griffiths et al.NeurIPS 2022 · 30 citations
  2. ContextRef: Evaluating Referenceless Metrics for Image Description GenerationElisa Kreiss, Eric Zelikman, Christopher Potts, Nick HaberICLR 2024 · 6 citations
  3. Dealing with Semantic Underspecification in Multimodal NLPSandro PezzelleACL 2023 · 5 citations

Related papers

  1. Journalistic Guidelines Aware News Image CaptioningXuewen Yang, Svebor Karaman, Joel R. Tetreault, Alejandro JaimesEMNLP 2021
  2. Aesthetically Relevant Image CaptioningZhipeng Zhong, Fei Zhou, Guoping QiuAAAI 2023 · 16 citations
  3. Query-based Image Captioning from Multi-context 360cdegree ImagesKoki Maeda, Shuhei Kurita, Taiki Miyanishi, Naoaki OkazakiEMNLP Findings 2023 · 1 citation
  4. ICECAP: Information Concentrated Entity-aware Image CaptioningAnwen Hu, Shizhe Chen, Qin JinACM MM 2020 · 20 citations
  5. CARIS: Context-Aware Referring Image SegmentationSun'ao Liu, Yiheng Zhang, Zhaofan Qiu, Hongtao Xie et al.ACM MM 2023 · 34 citations
Dusk over the sea between two cliffs drawn in fine vertical lines