EnvPipe: Performance-preserving DNN Training Framework for Saving Energy

Energy saving is a crucial mission for data center providers. Among many services, DNN training and inference are significant contributors to energy consumption. This work focuses on saving energy in multi-GPU DNN training. Typically, energy savings come at the cost of some degree of performance degradation. However, determining the acceptable level of performance degradation for a long-running training job can be difficult.

This work proposes ENVPIPE, an energy-saving DNN training framework. ENVPIPE aims to maximize energy saving while maintaining negligible performance slowdown. EN-VPIPE takes advantage of slack time created by bubbles in pipeline parallelism. It schedules pipeline units to place bubbles after pipeline units as frequently as possible and then stretches the execution time of pipeline units by lowering the SM frequency. During this process, ENVPIPE does not modify hyperparameters or pipeline dependencies, preserving the original accuracy of the training task. It selectively lowers the SM frequency of pipeline units to avoid performance degradation. We implement ENVPIPE as a library using PyTorch and demonstrate that it can save up to 25.2% energy in singlenode training with 4 GPUs and 28.4% in multi-node training with 16 GPUs, while keeping performance degradation to less than 1%. This design idea is generally applicable to any DNN training job where layer-wise partitioning is feasible, enabling training with pipeline parallelism across a large number of GPUs. However, realizing the design idea is challenging due to the following problems. i) To decide the value of SM frequency, ENVPIPE needs to know the trend of clock speed and training performance. The trend varies according to GPU hardware, batch size, and the number of layers in a pipeline stage. Offline profiling to obtain this information is impractical. Instead, ENVPIPE performs online profiling by sweeping SM frequencies to obtain the energy-saving curve under the given hyperparameters and GPU hardware. ii) How to schedule pipeline units decides the amount of bubbles that can be exploited. It is essential to ensure that a bubble exists after a pipeline unit of which execution time is stretched while ensuring that the next unit, which has data dependency, is sufficiently distant to avoid overall performance degradation.

ENVPIPE is implemented as a library using the existing ML framework. The ENVPIPE policy and mechanism are clearly separated, so developers can easily add required APIs to support a new ML framework. The current prototype of ENVPIPE is implemented on the DeepSpeed [25] library and uses the existing GPU device driver to control SM frequency. We evaluate ENVPIPE in real-world workloads: BERT, GPT, Megatron, and ResNet, and demonstrate the performance and energy saving on two GPU hardware: V100 and RTX3090. We perform evaluations of the workload in a single node (4 GPUs) and in multiple nodes (16 GPUs) and show that EN-VPIPE saves energy up to 25.2% and 28.4% in single-node and multi-node setups respectively while keeping performance degradation to less than 1%.

This paper makes the following contributions: • We present the design of ENVPIPE, a performancepreserving energy-saving DNN training framework that supports distributed training across multiple GPUs.

• ENVPIPE preserves the original statistical efficiency by not modifying any user-provided hyperparameters and controls only the side-effect-free control knob.

• We demonstrate ENVPIPE saves up to 25.2% and 28.4% energy saving in single-and multi-node GPU servers with less than 1% performance degradation.