An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models

Multimodal large language models (MLLMs) fine-tuned with multimodal instruction datasets have demonstrated remarkable capabilities in multimodal tasks.However, fine-tuning all parameters of MLLMs has become challenging as they usually contain billions of parameters.To address this issue, we study parameter-efficient fine-tuning (PEFT) methods for MLLMs.We aim to identify effective methods for enhancing the performance of MLLMs in scenarios where only a limited number of parameters are trained.This paper conducts empirical studies using four popular PEFT methods to fine-tune the LLM component of opensource MLLMs.We present a comprehensive analysis that encompasses various aspects, including the impact of PEFT methods on various models, parameters and location of the PEFT module, size of fine-tuning data, model stability based on PEFT methods, MLLM's generalization, and hallucination.We evaluated four PEFT methods on seven datasets from two different categories: unseen and seen datasets.Across all experiments, we show that the adapter is the best-performing PEFT method.At the same time, fine-tuning the connector layers leads to improved performance in most MLLMs.Code and data are available at https://github.com/alenai97/PEFT-MLLM.git