DeMIC: Decentralized Meta In-Context Learning with Refiner-Guided Adaptation

In-Context Learning (ICL) empowers Large Language Models to adapt to novel tasks, presenting a promising solution for privacy-preserving distributed scenarios. However, relying exclusively on inference-time ICL suffers from inherent instability. While fine-tuning models on specific tasks could mitigate these issues, as high-quality data is inherently dispersed across private agents, centralized fine-tuning becomes infeasible due to strict privacy constraints. To bridge this gap, we propose DeMIC, a novel decentralized meta-learning framework designed to collaboratively optimize the intrinsic in-context adaptation mechanism under task heterogeneity and dynamic network topologies. To overcome the insufficient exploration of standard meta-updates, DeMIC incorporates a Refiner-Guided mechanism. By constructing contrastive pairs, this mechanism forces the model to explore broader parameter spaces and learn from negative samples. Theoretically, we establish the first convergence guarantee for distributed Meta-ICL that explicitly models the context-driven adaptation process. Extensive experiments demonstrate significant generalization of DeMIC on new tasks, improving accuracy by 11.76% and F1-score by 10.65% on average over all settings.