Glasses: Enabling Fast Environment-aware Few-Shot Learning via Device-Cloud Collaboration

Pre-trained visual models have been deployed on various edge devices to facilitate a broad range of downstream tasks through few-shot learning (FSL), in particular when downstream data or on-device resources are limited. However, FSL often suffers from poor performance due to its inability to adapt to the characteristics of the deployment environments, while backbone fine-tuning prior to model deployment is typically infeasible because of the unavailability of environment-specific samples. To tackle this challenge, this paper presents Glasses, a lightweight fine-tuning scheme that can adapt ViT-based model backbones to deployment environments rapidly through device-cloud collaboration, helping the model achieve better FSL performance on the device. Glasses leverages the computational power and sample resources in the cloud to produce model updates for rapid model adaptation based on only one environment image without needing a label. Experiments with five models on two datasets demonstrate that Glasses can adapt a model rapidly and outperform the original backbone by 5.54%–22.56% in the 1-shot setting and by 2.50%–10.28% in the 5-shot setting. The source code is available at https://github.com/CGCL-codes/Glasses.