Lune

ACL Findings2023

Contrastive Training Improves Zero-Shot Classification of Semi-structured Documents

Muhammad Khalifa, Yogarshi Vyas, Shuai Wang, Graham Horwood, Sunil Mallya, Miguel Ballesteros

Abstract

We investigate semi-structured document classification in a zero-shot setting. Classification of semi-structured documents is more challenging than that of standard unstructured documents, as positional, layout, and style information play a vital role in interpreting such documents. The standard classification setting where categories are fixed during both training and testing falls short in dynamic environments where new document categories could potentially emerge. We focus exclusively on the zero-shot setting where inference is done on new unseen classes. To address this task, we propose a matching-based approach that relies on a pairwise contrastive objective for both pretraining and fine-tuning. Our results show a significant boost in Macro F 1 from the proposed pretraining step in both supervised and unsupervised zero-shot settings.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  • LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking cites DocFormer: End-to-End Transformer for Document Understanding
  • DocFormer: End-to-End Transformer for Document Understanding cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  • DocFormer: End-to-End Transformer for Document Understanding cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. DeAL: Decoding-time Alignment for Large Language ModelsJames Y. Huang, Sailik Sengupta, Daniele Bonadiman, Yi-An Lai et al.ACL 2025

Builds on10

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. LayoutLMv3: Pre-training for Document AI with Unified Text and Image MaskingYupan Huang, Tengchao Lv, Lei Cui, Yutong Lu et al.ACM MM 2022 · 606 citations
  3. LayoutLM: Pre-training of Text and Layout for Document Image UnderstandingYiheng Xu, Minghao Li, Lei Cui, Shaohan Huang et al.KDD 2020 · 575 citations
  4. DocFormer: End-to-End Transformer for Document UnderstandingSrikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie et al.ICCV 2021 · 392 citations
  5. Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection NetworkYutai Hou, Wanxiang Che, Yongkui Lai, Zhihan Zhou et al.ACL 2020 · 186 citations

Related papers

  1. SemSup-XC: Semantic Supervision for Zero and Few-shot Extreme ClassificationPranjal Aggarwal, Ameet Deshpande, Karthik R. NarasimhanICML 2023 · 8 citations
  2. Semantic matching for text classification with complex class descriptionsBrian de Silva, Kuan-Wen Huang, Gwang Lee, Karen Hovsepian et al.EMNLP 2023 · 1 citation
  3. Zero-Shot Object Detection by Semantics-Aware DETR with Adaptive Contrastive LossHuan Liu, Lu Zhang, Jihong Guan, Shuigeng ZhouACM MM 2023 · 6 citations
  4. Liberating Seen Classes: Boosting Few-Shot and Zero-Shot Text Classification via Anchor Generation and Classification ReframingHan Liu, Siyang Zhao, Xiaotong Zhang, Feng Zhang et al.AAAI 2024 · 7 citations
  5. PESCO: Prompt-enhanced Self Contrastive Learning for Zero-shot Text ClassificationYau-Shian Wang, Ta-Chung Chi, Ruohong Zhang, Yiming YangACL 2023 · 17 citations
Dusk over the sea between two cliffs drawn in fine vertical lines