ACL Findings2023
Contrastive Training Improves Zero-Shot Classification of Semi-structured Documents
Muhammad Khalifa, Yogarshi Vyas, Shuai Wang, Graham Horwood, Sunil Mallya, Miguel Ballesteros
Abstract
We investigate semi-structured document classification in a zero-shot setting. Classification of semi-structured documents is more challenging than that of standard unstructured documents, as positional, layout, and style information play a vital role in interpreting such documents. The standard classification setting where categories are fixed during both training and testing falls short in dynamic environments where new document categories could potentially emerge. We focus exclusively on the zero-shot setting where inference is done on new unseen classes. To address this task, we propose a matching-based approach that relies on a pairwise contrastive objective for both pretraining and fine-tuning. Our results show a significant boost in Macro F 1 from the proposed pretraining step in both supervised and unsupervised zero-shot settings.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking cites DocFormer: End-to-End Transformer for Document Understanding
- DocFormer: End-to-End Transformer for Document Understanding cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
- DocFormer: End-to-End Transformer for Document Understanding cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers1
Builds on10
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image MaskingYupan Huang, Tengchao Lv, Lei Cui, Yutong Lu et al.ACM MM 2022 · 606 citations
- LayoutLM: Pre-training of Text and Layout for Document Image UnderstandingYiheng Xu, Minghao Li, Lei Cui, Shaohan Huang et al.KDD 2020 · 575 citations
- DocFormer: End-to-End Transformer for Document UnderstandingSrikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie et al.ICCV 2021 · 392 citations
- Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection NetworkYutai Hou, Wanxiang Che, Yongkui Lai, Zhihan Zhou et al.ACL 2020 · 186 citations
Related papers
- SemSup-XC: Semantic Supervision for Zero and Few-shot Extreme ClassificationPranjal Aggarwal, Ameet Deshpande, Karthik R. NarasimhanICML 2023 · 8 citations
- Semantic matching for text classification with complex class descriptionsBrian de Silva, Kuan-Wen Huang, Gwang Lee, Karen Hovsepian et al.EMNLP 2023 · 1 citation
- Zero-Shot Object Detection by Semantics-Aware DETR with Adaptive Contrastive LossHuan Liu, Lu Zhang, Jihong Guan, Shuigeng ZhouACM MM 2023 · 6 citations
- Liberating Seen Classes: Boosting Few-Shot and Zero-Shot Text Classification via Anchor Generation and Classification ReframingHan Liu, Siyang Zhao, Xiaotong Zhang, Feng Zhang et al.AAAI 2024 · 7 citations
- PESCO: Prompt-enhanced Self Contrastive Learning for Zero-shot Text ClassificationYau-Shian Wang, Ta-Chung Chi, Ruohong Zhang, Yiming YangACL 2023 · 17 citations
