PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning

As multimedia content expands, the demand for unified multimodal retrieval (UMR) in real-world applications increases. Recent work leverages multimodal large language models to tackle this task. However, the large number of parameters leads to high training resource demands and low inference efficiency. To address this issue, we propose the PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning, an efficient approach to enhancing the unified retrieval capabilities from both structure and learning perspectives: 1) From the perspective of model structure, to retain the most retrieval-relevant components within MLLMs, we analyze and propose Layer-Pruned Self-Distillation approach. It structurally prunes the model by preserving only the shallow layers, substantially reducing the parameters of MLLM. Moreover, we use self-distillation to mitigate the representational degradation caused by pruning. It reuses the feature from dropped deep layers as the teacher signal, where the supervised signal enables the retrieval embedding token to efficiently inherit effective representational capacity, resulting in a more compact model. 2) From the perspective of model learning, to mitigate representation degradation caused by rapid convergence during multimodal contrastive learning, we propose Modality-Adaptive Contrastive Learning Loss (MAC-Loss). It adaptively separates in-batch negative candidate samples into harder intramodality and simpler inter-modality groups based on each query's target modality. Assigning each group a temperature coefficient with different strategies enables each query to adaptively focus on challenging in-batch negatives, reducing the resource demands of multimodal contrastive learning. Experiments demonstrate that our approach achieves double efficiency, significantly reduces resource consumption while maintaining most of the performance.