Lune

CVPR2024Top-tier venue

X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization

Anna Kukleva, Fadime Sener, Edoardo Remelli, Bugra Tekin, Eric Sauser, Bernt Schiele, Shugao Ma

Abstract

Lately, there has been growing interest in adapting vision-language models (VLMs) to image and third-person video classification due to their success in zero-shot recognition. However, the adaptation of these models to egocentric videos has been largely unexplored. To address this gap, we propose a simple yet effective cross-modal adaptation framework, which we call X-MIC. Using a video adapter, our pipeline learns to align frozen text embeddings to each egocentric video directly in the shared embedding space. Our novel adapter architecture retains and improves generalization of the pre-trained VLMs by disentangling learnable temporal modeling and frozen visual encoder. This results in an enhanced alignment of text embeddings to each egocentric video, leading to a significant improvement in cross-dataset generalization. We evaluate our approach on the Epic-Kitchens, Ego4D, and EGTEA datasets for fine-grained cross-dataset action generalization, demonstrating the effectiveness of our method. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Conditional Prompt Learning for Vision-Language Models cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • Conditional Prompt Learning for Vision-Language Models cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
  • Conditional Prompt Learning for Vision-Language Models cites Learning Transferable Visual Models From Natural Language Supervision
  • EgoPrompt: Prompt Learning for Egocentric Action Recognition cites Conditional Prompt Learning for Vision-Language Models
  • EgoPrompt: Prompt Learning for Egocentric Action Recognition cites Learning Transferable Visual Models From Natural Language Supervision
  • AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition cites Learning Transferable Visual Models From Natural Language Supervision
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. EgoPrompt: Prompt Learning for Egocentric Action RecognitionHuaihai Lyu, Chaofan Chen, Yuheng Ji, Changsheng XuACM MM 2025 · 3 citations
  2. SAVA-X: Ego-to-Exo Imitation Error Detection via Scene-Adaptive View Alignment and Bidirectional Cross View FusionXiang Li, Heqian Qiu, Lanxiao Wang, Benliu Qiu et al.CVPR 2026 · 1 citation
  3. Pro 2 Assist: Continuous Step-aware Proactive Assistance with Multi-modal Egocentric Perception for Long-horizon Procedural TasksLilin Xu, Bufang Yang, Siyang Jiang, Kaiwei Liu et al.UbiComp 2026
  4. Context-Enhanced Memory-Refined Transformer for Online Action DetectionZhanzhong Pang, Fadime Sener, Angela YaoCVPR 2025
  5. Ego3S: Select, Strengthen, and Synchronize for Efficient Egocentric ReasoningShenshen Li, Kaiyuan Deng, RuoHuai Xie, Xing Xu et al.ICML 2026

Builds on17

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
  3. Conditional Prompt Learning for Vision-Language ModelsKaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei LiuCVPR 2022 · 1,438 citations
  4. AdaptFormer: Adapting Vision Transformers for Scalable Visual RecognitionShoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang et al.NeurIPS 2022 · 1,291 citations
  5. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations

Related papers

  1. MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language KnowledgeWei Lin, Leonid Karlinsky, Nina Shvetsova, Horst Possegger et al.ICCV 2023 · 52 citations
  2. VTD-CLIP: Video-to-Text Discretization via Prompting CLIPWencheng Zhu, Yuexin Wang, Hongxuan Li, Pengfei ZhuAAAI 2026 · 2 citations
  3. HomoGraphAdapter: A Homogeneous Graph Neural Network as an Effective Adapter for Vision-Language ModelsChuan He, Zhuozhao Li, Song Guo, Xiaocheng Lu et al.EMNLP Findings 2025
  4. A Multimodal, Multi-Task Adapting Framework for Video Action RecognitionMengmeng Wang, Jiazheng Xing, Boyuan Jiang, Jun Chen et al.AAAI 2024 · 13 citations
  5. ORION: ORthonormal Text Encoding for Universal VLM AdaptatIONOmprakash Chakraborty, Jose Dolz, Ismail Ben AyedCVPR 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines