Image Similarity Using an Ensemble of Context-Sensitive Models
Abstract
Image similarity has been extensively studied in computer vision. In recent years, machine-learned models have shown their ability to encode more semantics than traditional multivariate metrics. However, in labelling semantic similarity, assigning a numerical score to a pair of images is impractical, making the improvement and comparisons on the task difficult. In this work, we present a more intuitive approach to build and compare image similarity models based on labelled data in the form of A:R vs B:R, i.e., determining if an image A is closer to a reference image R than another image B. We address the challenges of sparse sampling in the image space (R, A, B) and biases in the models trained with context-based data by using an ensemble model. Our testing results show that the ensemble model constructed performs ∼ 5% better than the best individual context-sensitive models. They also performed better than the models that were directly fine-tuned using mixed imagery data as well as existing deep embeddings, e.g., CLIP [30] and DINO [3]. This work demonstrates that context-based labelling and model training can be effective when an appropriate ensemble approach is used to alleviate the limitation due to sparse sampling.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data cites High-Resolution Image Synthesis with Latent Diffusion Models
- DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data cites Emerging Properties in Self-Supervised Vision Transformers
- DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Emerging Properties in Self-Supervised Vision Transformers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Background Invariance Testing According to Semantic Proximity cites DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data
- Background Invariance Testing According to Semantic Proximity cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Cited by top-tier papers1
Builds on9
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Emerging Properties in Self-Supervised Vision TransformersMathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou et al.ICCV 2021 · 8,921 citations
- DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic DataStephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai et al.NeurIPS 2023 · 413 citations
Related papers
- Unsupervised Clustering using Pseudo-semi-supervised LearningDivam Gupta, Ramachandran Ramjee, Nipun Kwatra, Muthian SivathanuICLR 2020 · 21 citations
- Relational Visual SimilarityThao Nguyen, Sicheng Mo, Krishna Kumar Singh, Yilin Wang et al.CVPR 2026 · 1 citation
- Supervised Metric Learning to Rank for Retrieval via Contextual Similarity OptimizationChristopher Liao, Theodoros Tsiligkaridis, Brian KulisICML 2023 · 10 citations
- Looking Beyond Single Images for Contrastive Semantic Segmentation LearningFeihu Zhang, Philip H. S. Torr, René Ranftl, Stephan R. RichterNeurIPS 2021 · 44 citations
- Semantic-Enhanced Image ClusteringShaotian Cai, Liping Qiu, Xiaojun Chen, Qin Zhang et al.AAAI 2023 · 52 citations
