ACL Findings2021
Transformer-Exclusive Cross-Modal Representation for Vision and Language
Abstract
Ever since the advent of deep learning, crossmodal representation learning has been dominated by the approaches involving convolutional neural networks for visual representation and recurrent neural networks for language representation. Transformer architecture, however, has rapidly taken over the recurrent neural networks in natural language processing tasks, and it has also been shown that vision tasks can be handled with transformer architecture, with compatible performance to convolutional neural networks. Such results naturally lead to speculation upon the possibility of tackling cross-modal representation for vision and language exclusively with transformer. This paper examines transformerexclusive cross-modal representation to explore such possibility, demonstrating its potentials as well as discussing its current limitations and its prospects.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- UniT: Multimodal Multitask Learning with a Unified Transformer cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- UniT: Multimodal Multitask Learning with a Unified Transformer cites Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training
- UniT: Multimodal Multitask Learning with a Unified Transformer cites Training data-efficient image transformers & distillation through attention
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision cites Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision cites Training data-efficient image transformers & distillation through attention
- Training data-efficient image transformers & distillation through attention cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds on6
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Training data-efficient image transformers & distillation through attentionHugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa et al.ICML 2021 · 8,974 citations
- ViLT: Vision-and-Language Transformer Without Convolution or Region SupervisionWonjae Kim, Bokyung Son, Ildoo KimICML 2021 · 2,258 citations
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-TrainingGen Li, Nan Duan, Yuejian Fang, Ming Gong et al.AAAI 2020 · 966 citations
- UniT: Multimodal Multitask Learning with a Unified TransformerRonghang Hu, Amanpreet SinghICCV 2021 · 354 citations
Related papers
- TVLT: Textless Vision-Language TransformerZineng Tang, Jaemin Cho, Yixin Nie, Mohit BansalNeurIPS 2022 · 40 citations
- Vision-Language Models Create Cross-Modal Task RepresentationsGrace Luo, Trevor Darrell, Amir BarICML 2025
- Unifying Two-Stream Encoders with Transformers for Cross-Modal RetrievalYi Bin, Haoxuan Li, Yahui Xu, Xing Xu et al.ACM MM 2023 · 33 citations
- LAVT: Language-Aware Vision Transformer for Referring Image SegmentationZhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen et al.CVPR 2022 · 319 citations
- Omnivore: A Single Model for Many Visual ModalitiesRohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten et al.CVPR 2022 · 185 citations
