Prototype-based Dataset Comparison
Abstract
Dataset summarisation is a fruitful approach to dataset inspection. However, when applied to a single dataset the discovery of visual concepts is restricted to those most prominent. We argue that a comparative approach can expand upon this paradigm to enable richer forms of dataset inspection that go beyond the most prominent concepts.To enable dataset comparison we present a module that learns concept-level prototypes across datasets. We leverage self-supervised learning to discover these prototypes without supervision, and we demonstrate the benefits of our approach in two case-studies. Our findings show that dataset comparison extends dataset inspection and we hope to encourage more works in this direction. Code and usage instructions available at https://github.com/Nanne/ProtoSim
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Emerging Properties in Self-Supervised Vision Transformers cites A Simple Framework for Contrastive Learning of Visual Representations
- Emerging Properties in Self-Supervised Vision Transformers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Emerging Properties in Self-Supervised Vision Transformers cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- Emerging Properties in Self-Supervised Vision Transformers cites Training data-efficient image transformers & distillation through attention
- ConceptScope: Characterizing Dataset Bias via Disentangled Visual Concepts cites Describing Differences in Image Sets with Natural Language
- ConceptScope: Characterizing Dataset Bias via Disentangled Visual Concepts cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images cites Describing Differences in Image Sets with Natural Language
- Training data-efficient image transformers & distillation through attention cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge cites Emerging Properties in Self-Supervised Vision Transformers
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- ImageSet2Text: Describing Sets of Images Through Text cites Describing Differences in Image Sets with Natural Language
- ImageSet2Text: Describing Sets of Images Through Text cites Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images
Cited by top-tier papers6
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual KnowledgeFawaz Sammani, Nikos DeligiannisNeurIPS 2024 · 15 citations
- ConceptScope: Characterizing Dataset Bias via Disentangled Visual ConceptsJinho Choi, Hyesu Lim, Steffen Schneider, Jaegul ChooNeurIPS 2025 · 5 citations
- Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of ImagesBoyang Deng, Songyou Peng, Kyle Genova, Gordon Wetzstein et al.ICCV 2025 · 4 citations
- ImageSet2Text: Describing Sets of Images Through TextPiera Riccio, Francesco Galati, Kajetan Schweighofer, Noa Garcia et al.AAAI 2026 · 1 citation
- Describing Differences in Image Sets with Natural LanguageLisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong et al.CVPR 2024
Builds on17
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec et al.NeurIPS 2020 · 9,171 citations
- Training data-efficient image transformers & distillation through attentionHugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa et al.ICML 2021 · 8,974 citations
- Emerging Properties in Self-Supervised Vision TransformersMathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou et al.ICCV 2021 · 8,921 citations
Related papers
- CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud UnderstandingMohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri et al.CVPR 2022 · 286 citations
- Dataset Ownership Verification in Contrastive Pre-trained ModelsYuechen Xie, Jie Song, Mengqi Xue, Haofei Zhang et al.ICLR 2025
- Coreset Sampling from Open-Set for Fine-Grained Self-Supervised LearningSungnyun Kim, Sangmin Bae, Se-Young YunCVPR 2023
- When Does Contrastive Visual Representation Learning Work?Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha et al.CVPR 2022 · 98 citations
- Prototypical Cross-domain Knowledge Transfer for Cervical Dysplasia Visual InspectionYichen Zhang, Yifang Yin, Ying Zhang, Zhenguang Liu et al.ACM MM 2023 · 4 citations
