Resilience-Aware Elastic Scaling for Cloud-Native Online DL Training on Multi-Tenant GPU Clusters

Online deep learning (DL) training has become pivotal in powering real-time applications. Yet tidal workload fluctuations leave GPU clusters significantly underutilized during off-peak periods. This imbalance not only wastes GPU capacity but also exacerbates scarcity for other GPU-intensive jobs on cloud-native GPU cluster. Cluster-wide resource leasing across different tenants enabled by elastic scaling offers a promising opportunity to enhance GPU utilization for cloud-native online DL training deployments in multi-tenant GPU clusters. Existing solutions do not address the unique challenges associated with maintaining system stability during elastic scaling for online DL training jobs, including prolonged disruptions due to job reconstruction, failures arising from dependency-unaware operation triggering, and the unreliable reclamation of high-availability GPU resources. In this paper, we introduce WeFlex, a resilience-aware elastic scaling solution engineered for cloud-native online deep learning jobs in multi-tenant GPU clusters. WeFlex enables online training jobs to lease idle GPUs for other GPU-intensive jobs during low-demand periods while ensuring rapid reclamation as demand surges. It significantly reduces the duration of training disruptions through constructing a interruption mitigation pipeline, prevents dependency-unaware operation failures via topology-aware pod orchestration, and ensure reclamation of high-availability GPU resources through right-of-return GPU leasing. Evaluations on 10,000-plus scale GPU clusters in production demonstrate that WeFlex significantly enhances GPU utilization while reliably maintaining continuous training performance.