Unsupervised Anomaly Detection in Multivariate Time Series across Heterogeneous Domains

The widespread adoption of digital services, along with the scale and complexity at which they operate, has made incidents in IT operations increasingly more likely, diverse, and impactful. This has led to the rapid development of a central aspect of "Artificial Intelligence for IT Operations" (AIOps), focusing on detecting anomalies in vast amounts of multivariate time series data generated by service entities. In this paper, we begin by introducing a unifying framework for benchmarking unsupervised anomaly detection (AD) methods, and highlight the problem of shifts in normal behaviors that can occur in practical AIOps scenarios. To tackle anomaly detection under domain shift, we then cast the problem in the framework of domain generalization and propose a novel approach, Domain-Invariant VAE for Anomaly Detection (DIVAD), to learn domain-invariant representations for unsupervised anomaly detection. Our evaluation results using the Exathlon benchmark show that the two main DIVAD variants significantly outperform the best unsupervised AD method in maximum performance, with 20% and 15% improvements in maximum peak F1-scores, respectively. Evaluation using the Application Server Dataset further demonstrates the broader applicability of our domain generalization methods. * The number of records in the last received batch. -5 block manager features. For example: * The disk space used by the block manager. * The memory used by the block manager. -32 JVM features. For example: * The heap memory usage of the driver. * The survivor space usage of the driver (the survivor space is a memory pool that holds objects having survived a young generation garbage collection, before those objects potentially get promoted to old generation memory). -19 DAG scheduler features. For example: * The number of active jobs. * The number of running stages. -94 live listener bus features. For example: * The number of messages received from the DAG scheduler in the last 1, 5 and 15 minutes. * The average processing time of messages received from the DAG scheduler. • 69 Executor Features (Averaged Across Active Execs) -27 "executor" features. For example: * The CPU time. * The number of active tasks. * The number of bytes read and written to HDFS. -38 JVM features, similar to those of the driver. -4 netty block transfer features. For example: * The direct memory used by the shuffle client and server of the netty network application framework (sending and receiving blocks of data). * The heap memory used by the shuffle client and server of the netty network application framework.