X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization
Anna Kukleva, Fadime Sener, Edoardo Remelli, Bugra Tekin, Eric Sauser, Bernt Schiele, Shugao Ma
Abstract
Lately, there has been growing interest in adapting vision-language models (VLMs) to image and third-person video classification due to their success in zero-shot recognition. However, the adaptation of these models to egocentric videos has been largely unexplored. To address this gap, we propose a simple yet effective cross-modal adaptation framework, which we call X-MIC. Using a video adapter, our pipeline learns to align frozen text embeddings to each egocentric video directly in the shared embedding space. Our novel adapter architecture retains and improves generalization of the pre-trained VLMs by disentangling learnable temporal modeling and frozen visual encoder. This results in an enhanced alignment of text embeddings to each egocentric video, leading to a significant improvement in cross-dataset generalization. We evaluate our approach on the Epic-Kitchens, Ego4D, and EGTEA datasets for fine-grained cross-dataset action generalization, demonstrating the effectiveness of our method. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Conditional Prompt Learning for Vision-Language Models cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Conditional Prompt Learning for Vision-Language Models cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
- Conditional Prompt Learning for Vision-Language Models cites Learning Transferable Visual Models From Natural Language Supervision
- EgoPrompt: Prompt Learning for Egocentric Action Recognition cites Conditional Prompt Learning for Vision-Language Models
- EgoPrompt: Prompt Learning for Egocentric Action Recognition cites Learning Transferable Visual Models From Natural Language Supervision
- AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition cites Learning Transferable Visual Models From Natural Language Supervision
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers5
- EgoPrompt: Prompt Learning for Egocentric Action RecognitionHuaihai Lyu, Chaofan Chen, Yuheng Ji, Changsheng XuACM MM 2025 · 3 citations
- SAVA-X: Ego-to-Exo Imitation Error Detection via Scene-Adaptive View Alignment and Bidirectional Cross View FusionXiang Li, Heqian Qiu, Lanxiao Wang, Benliu Qiu et al.CVPR 2026 · 1 citation
- Pro 2 Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural TasksLilin Xu, Bufang Yang, Siyang Jiang, Kaiwei Liu et al.UbiComp 2026
- Context-Enhanced Memory-Refined Transformer for Online Action DetectionZhanzhong Pang, Fadime Sener, Angela YaoCVPR 2025
- Ego3S: Select, Strengthen, and Synchronize for Efficient Egocentric ReasoningShenshen Li, Kaiyuan Deng, RuoHuai Xie, Xing Xu et al.ICML 2026
Builds on17
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
- Conditional Prompt Learning for Vision-Language ModelsKaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei LiuCVPR 2022 · 1,438 citations
- AdaptFormer: Adapting Vision Transformers for Scalable Visual RecognitionShoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang et al.NeurIPS 2022 · 1,291 citations
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
Related papers
- MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language KnowledgeWei Lin, Leonid Karlinsky, Nina Shvetsova, Horst Possegger et al.ICCV 2023 · 52 citations
- VTD-CLIP: Video-to-Text Discretization via Prompting CLIPWencheng Zhu, Yuexin Wang, Hongxuan Li, Pengfei ZhuAAAI 2026 · 2 citations
- HomoGraphAdapter: A Homogeneous Graph Neural Network as an Effective Adapter for Vision-Language ModelsChuan He, Zhuozhao Li, Song Guo, Xiaocheng Lu et al.EMNLP Findings 2025
- A Multimodal, Multi-Task Adapting Framework for Video Action RecognitionMengmeng Wang, Jiazheng Xing, Boyuan Jiang, Jun Chen et al.AAAI 2024 · 13 citations
- ORION: ORthonormal Text Encoding for Universal VLM AdaptatIONOmprakash Chakraborty, Jose Dolz, Ismail Ben AyedCVPR 2026 · 1 citation
