ArrayPipe: Introducing Job-Array Pipeline Parallelism for High Throughput Model Exploration

Deep Learning (DL) applications have experienced exponential growth in data volume and model complexity, spurring various parallel approaches. Existing solutions mostly focus on accelerating individual training jobs. However, jobs submitted to a cluster may not always be independent. This is due to the distinctive characteristic of DL training that it is an exploratory process. Model developers often launch multiple training instances in a batch with the same model structure but different settings to tune hyper-parameters, which provides an opportunity to regard these jobs as job-arrays. With further support of low-cost job context switching, sharing resources among these jobs is not just feasible but also beneficial to the resource utilization and the throughput of a DL cluster. This paper introduces Job-Array Pipeline Parallelism (JAP) that assembles a batch of sibling DL training jobs into a concurrent job-array. We design ArrayPipe, a framework that supports high throughput model exploration with JAP. A novel scheduling problem in JAP is proposed that seeks to minimize the per-iteration training time for a job-array, along with two scheduling algorithms for different scales of job-arrays. Extensive testbed experiments and trace-driven simulations show that ArrayPipe achieves 1.46× training throughput on average compared with state-of-the-art related works.