Formally Exploring Visual Anomaly Detection Evaluation Metrics

Inaccurate Visual Anomaly Detection (VAD) can lead to critical failures in safety-sensitive domains, including autonomous navigation and industrial surveillance. Despite the abundance of VAD algorithms, assessing their capabilities remains challenging, because standard evaluation metrics often yield inconsistent or misleading results. In this paper, we demonstrate that standard evaluation paradigms fail to adequately capture model performance, often overlooking critical errors such as redundant detections and false positive distributions. To address this, we formalize the requirements for VAD evaluation by introducing a set of verifiable properties of evaluation methods. With a systematic analysis of current state-of-the-art evaluation methods, we prove that none satisfy the full suite of proposed properties, revealing significant, inherent inconsistencies. To bridge this gap, we introduce SAAM-ALARM, a novel evaluation metric mathematically designed to satisfy these formal properties. Our results show that SAAM-ALARM provides a more nuanced and theoretically sound assessment, offering a superior standard for performance benchmarking in anomaly detection.