MMAligner: Safeguarding Multimodal Large Language Models through Representation Calibration

Multimodal large language models (MLLMs) have demonstrated impressive capabilities across a wide range of generative tasks. However, introducing non-text modalities poses significant challenges for safety alignment. MLLMs often refuse unsafe text-only prompts while producing harmful responses to semantically equivalent multimodal inputs. Existing mitigation strategies, including external guardrails and safety-oriented fine-tuning, largely overlook the mechanisms driving this safety disparity. External guardrails merely circumvent the model's intrinsic defects, while safety-oriented finetuning treats alignment as a black-box optimization problem, failing to diagnose and repair this flaw specifically. As a result, these methods either incur substantial inference latency with limited protection or significantly compromise model utility and rely heavily on large-scale multimodal datasets. Consequently, achieving effective and utility-preserving safety alignment for MLLMs remains an unresolved challenge. In this paper, we conduct a geometric analysis of MLLM representations to investigate the causes of multimodal safety degradation. We reveal that the safety mechanisms learned in the text-only modality actually persist in the multimodal setting. The safety subspace with refusal boundary remains valid across different modalities, where representations falling inside this boundary consistently elicit safe refusal responses. However, we observe a critical shift in representation where most unsafe multimodal inputs fall outside the boundary and bypass this intrinsic mechanism. This finding identifies the root cause of safety failure in the multimodal setting as a representation shift rather than a lack of safety capability. Motivated by this observation, we propose MMAligner, an MLLM safeguarding method based on representation calibration. Unlike existing methods, MMAligner addresses this representation shift by optimizing the model to map unsafe multimodal representations inside the pre-existing refusal boundary. Specifically, it adopts a hard lower bound to ensure refusal and a soft upper bound to prevent excessive modification, while preserving the representation of benign inputs. Extensive experiments * Qian Wang is the corresponding author. on multiple open-source MLLMs demonstrate that MMAligner increases the average refusal rate of multimodal unsafe inputs to 99% while incurring less than 2% utility degradation using minimal data, significantly outperforming existing baselines in the safety-utility trade-off. CCS Concepts • Security and privacy → Software and application security.