Lune

EMNLP Findings2023

Show, Write, and Retrieve: Entity-aware Article Generation and Retrieval

Zhongping Zhang, Yiwen Gu, Bryan A. Plummer

Abstract

Article comprehension is an important challenge in natural language processing with many applications such as article generation or imageto-article retrieval. Prior work typically encodes all tokens in articles uniformly using pretrained language models. However, in many applications, such as understanding news stories, these articles are based on real-world events and may reference many named entities that are difficult to accurately recognize and predict by language models. To address this challenge, we propose an ENtity-aware article GeneratIoN and rEtrieval (ENGINE) framework, to explicitly incorporate named entities into language models. ENGINE has two main components: a named-entity extraction module to extract named entities from both metadata and embedded images associated with articles, and an entity-aware mechanism that enhances the model's ability to recognize and predict entity names. We conducted experiments on three public datasets: GoodNews, VisualNews, and WikiText, where our results demonstrate that our model can boost both article generation and article retrieval performance, with a 4-5 perplexity improvement in article generation and a 3-4% boost in recall@1 in article retrieval. We release our implementation at this http URL.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites The Curious Case of Neural Text Degeneration
  • Language Models are Few-Shot Learners cites The Curious Case of Neural Text Degeneration
  • Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on10

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  3. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  4. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations
  5. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations

Related papers

  1. Visual News: Benchmark and Challenges in News Image CaptioningFuxiao Liu, Yinghan Wang, Tianlu Wang, Vicente OrdonezEMNLP 2021 · 67 citations
  2. ICECAP: Information Concentrated Entity-aware Image CaptioningAnwen Hu, Shizhe Chen, Qin JinACM MM 2020 · 20 citations
  3. Knowledge Completes the Vision: A Multimodal Entity-aware Retrieval-Augmented Generation Framework for News Image CaptioningXiaoxing You, Qiang Huang, Lingyu Li, Chi Zhang et al.AAAI 2026 · 1 citation
  4. Fine-tuning with Multi-modal Entity Prompts for News Image CaptioningJingjing Zhang, Shancheng Fang, Zhendong Mao, Zhiwei Zhang et al.ACM MM 2022 · 16 citations
  5. Show, Interpret and Tell: Entity-Aware Contextualised Image Captioning in WikipediaKhanh Nguyen, Ali Furkan Biten, Andrés Mafla, Lluís Gómez et al.AAAI 2023 · 15 citations
Dusk over the sea between two cliffs drawn in fine vertical lines