Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector

Deepfake detection is a long-established research topic vital for mitigating the spread of malicious misinformation. Unlike prior methods that provide either binary classification results or textual explanations separately, we introduce a novel method capable of generating both simultaneously. Our method harnesses the multi-modal learning capability of the pre-trained CLIP and the unprecedented interpretability of large language models (LLMs) to enhance both the generalization and explainability of deepfake detection. Specifically, we introduce a multi-modal face forgery detector (M2F2-Det) that employs tailored face forgery prompt learning, incorporating the pre-trained CLIP to improve generalization to unseen forgeries. Also, M2F2-Det incorporates an LLM to provide detailed textual explanations of its detection decisions, enhancing interpretability by bridging the gap between natural language and subtle cues of facial forgeries. Empirically, we evaluate M2F2-Det on both detection and explanation generation tasks, where it achieves state-of-the-art performance, demonstrating its effectiveness in identifying and explaining diverse forgeries. Source code is available at link. Recently, the powerful capability of vision-language models, e.g., CLIP [56], also inspired efforts in detecting deepfakes. For example, DDVQA-BLIP [85] reformulates deepfake detection as an explanation generation task using a vision-language model [37], which enhances interpretability through natural language descriptions (Fig. 1b ). In addition, several binary detectors [10, 52, 60 ] leverage CLIP's robust recognition capabilities to achieve impressive performance. However, three key limitations remain in these works. First, DDVQA-BLIP relies on a general text-generation model without dedicated mechanisms for deepfake detection, resulting in lower detection accuracy compared to conventional binary detectors. Secondly, prior CLIP-based detectors often lack effective input text prompts to describe diverse forgeries, restricting the adaptation of CLIP's multi-modal learning ability in the detection task. Third, while CLIP's open-set recognition capability -enabling it to identify diverse visual semantics -is successfully combined with LLMs in domains like document parsing [25, 48, 79] and medical diagnosis [34, 50, 83], its integration with LLMs for deepfake detection remains largely unexplored.