EMNLP Findings2024
Infrared-LLaVA: Enhancing Understanding of Infrared Images in Multi-Modal Large Language Models
Shixin Jiang, Zerui Chen, Jiafeng Liang, Yanyan Zhao, Ming Liu, Bing Qin
Abstract
Expanding the understanding capabilities of multi-modal large language models (MLLMs) for infrared modality is a challenge due to the single-modality nature and limited amount of training data. Existing methods typically construct a uniform embedding space for crossmodal alignment and leverage abundant visual image data to understand infrared images indirectly. However, they ignore the supervisory signals of infrared-modality-specific attributes, which may lead to a biased understanding of infrared images. To address this issue, we propose a debating multi-agent generation system that transfers knowledge from visible images to generate infrared image-text pairs and infrared instruction data. Moreover, we construct an infrared question-answering benchmark based on common infrared tasks. Experimental results from incremental fine-tuning on existing models and our Infrared-LLaVA-7B trained from scratch on infrared data demonstrate the effectiveness of the generated data and the feasibility of the generation approach.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting cites Visual Instruction Tuning
- IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting cites IRGPT: Understanding Real-World Infrared Image with Bi-Cross-Modal Curriculum on Large-Scale Benchmark
- Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels cites Visual Instruction Tuning
- Grounding Multimodal Large Language Models to the World cites Visual Instruction Tuning
- IRGPT: Understanding Real-World Infrared Image with Bi-Cross-Modal Curriculum on Large-Scale Benchmark cites Visual Instruction Tuning
Cited by top-tier papers3
- IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual PromptingTao Zhang, Yuyang Hong, Yang Xia, Kun Ding et al.CVPR 2026 · 8 citations
- TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared TranslationDong-Guw Lee, Tai Hyoung Rhee, Hyunsoo Jang, Young-Sik Shin et al.CVPR 2026 · 4 citations
- IRGPT: Understanding Real-World Infrared Image with Bi-Cross-Modal Curriculum on Large-Scale BenchmarkZhe Cao, Jin Zhang, Ruiheng ZhangICCV 2025 · 2 citations
Builds on12
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
- Grounding Multimodal Large Language Models to the WorldZhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao et al.ICLR 2024 · 1,170 citations
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent DebateChi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu et al.ICLR 2024 · 871 citations
- Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined LevelsHaoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen et al.ICML 2024 · 499 citations
Related papers
- Explore How to Inject Beneficial Noise in MLLMsRuishu Zhu, Sida Huang, Ziheng Jiao, Hongyuan ZhangAAAI 2026 · 7 citations
- VILA: On Pre-training for Visual Language ModelsJi Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov et al.CVPR 2024
- Aligning Large Multimodal Models with Factually Augmented RLHFZhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu et al.ACL Findings 2024 · 142 citations
- AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context LearningJun Gao, Qian Qiao, Tianxiang Wu, Zili Wang et al.AAAI 2025 · 12 citations
- UrbanLLaVA: A Multi-Modal Large Language Model for Urban IntelligenceJie Feng, Shengyuan Wang, Tianhui Liu, Yanxin Xi et al.ICCV 2025 · 7 citations
