Knowledge Transfer from Interaction Learning

Current visual foundation models (VFMs) face a fundamental limitation in transferring knowledge from vision language models (VLMs): while VLMs excel at modeling cross-modal interactions through unified representation spaces, existing VFMs predominantly adopt result-oriented paradigms that neglect the underlying interaction processes. This representational discrepancy hinders effective knowledge transfer and limits generalization across diverse vision tasks. We propose Learning from Interactions (LFI), a cognitive-inspired framework that addresses this gap by explicitly modeling visual understanding as an interactive process. Our key insight is that capturing the dynamic interaction patterns encoded in pre-trained VLMs - beyond their final representations - enables more faithful and efficient knowledge transfer to VFMs. The approach centers on two technical innovations: (1) Interaction Queries, which maintain persistent relational structures across network layers, and (2) interaction-based supervision, derived from the cross-modal attention mechanisms of VLMs. Comprehensive experiments demonstrate consistent improvements across multiple benchmarks: achieving $\sim 3.3 \%$ and +1.6 $m A P /+2.4 A P^{\text{mask }}$ absolute gains on TinyImageNet classification and COCO detection/segmentation respectively, with minimal parameter overhead and faster convergence $(7 \times$ speedup). The framework particularly excels in crossdomain settings, delivering $\sim 2.4 \%$ and $\sim 9.3 \%$ zero-shot improvements on PACS and VLCS. Human evaluations further confirm its cognitive alignment, outperforming resultoriented methods by $2.7 \times$ in semantic consistency metrics.