UP2ME: Univariate Pre-training to Multivariate Fine-tuning as a General-purpose Framework for Multivariate Time Series Analysis

Despite the success of self-supervised pre-training in texts and images, applying it to multivariate time series (MTS) falls behind tailored methods for tasks like forecasting, imputation and anomaly detection. We propose a general-purpose framework, named UP2ME (Univariate Pre-training to Multivariate Fine-tuning). It conducts taskagnostic pre-training when downstream tasks are unspecified. Once the task and setting (e.g. forecasting length) are determined, it gives sensible solutions with frozen pre-trained parameters, which has not been achieved before. UP2ME is further refined by fine-tuning. A univariate-tomultivariate paradigm is devised to address the heterogeneity of temporal and cross-channel dependencies. In univariate pre-training, univariate instances with diverse lengths are generated for Masked AutoEncoder (MAE) pre-training, discarding cross-channel dependency. The pretrained model handles downstream tasks by formulating them into specific mask-reconstruction problems. In multivariate fine-tuning, it constructs a dependency graph among channels using the pre-trained encoder to enhance cross-channel dependency capture. Experiments on eight realworld datasets show its SOTA performance in forecasting and imputation, approaching taskspecific performance in anomaly detection. Our code is available at https://github.com/ Thinklab-SJTU/UP2ME . Figure 1: UP2ME Workflow: Given the dataset, UP2ME performs task-agnostic univariate pre-training. The resulting pre-trained model can execute immediate forecasting, imputation and anomaly detection across various settings without parameter modifications. Once the downstream task and its setting are determined, multivariate fine-tuning tailors UP2ME to the specific task for more accurate solutions. UP2ME: a General-purpose Framework for Multivariate Time Series Analysis