Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation

CSR Rep. Sparse MRL Rep. Dense 0.1 0.4 0.2 0.3 (a) Overview of different representations (c) Training GPU hours of different models (b) Retrieval efficiency across different methods 0.7 0.3 0 0.1 0 0 0.9 Full Rep. Dense Feature Dim Figure 1. Overview of our proposed method. (a) Illustrative comparison between standard embeddings (dense, long) and two different compression schemes: Matryoshka representations (MRL) (Kusupati et al., 2022) with short length and our Contrastive Sparse Representation (CSR) based on sparsification. (b) Comparison of retrieval accuracy and time of different methods on ImageNet with GPU. Compared to MRL and int8 quantification (Quant Int8) methods, our sparse embedding approach CSR attains the best retrieval accuracy (very close to full representations) while being much more efficient in retrieval time, using sparse matrix multiplication on GPU. (c) Training GPU hours of CSR compared to baseline methods, where we outperform MRL on 1-NN accuracy with much less training time.