Accelerating neural network training: An analysis of the AlgoPerf competition

The goal of the ALGOPERF: TRAINING ALGORITHMS competition is to evaluate practical speed-ups in neural network training achieved solely by improving the underlying training algorithms. In the external tuning ruleset, submissions must provide workload-agnostic hyperparameter search spaces, while in the self-tuning ruleset they must be completely hyperparameter-free. In both rulesets, submissions are compared on time-to-result across multiple deep learning workloads, training on fixed hardware. This paper presents the inaugural ALGOPERF competition's results, which drew 18 diverse submissions from 10 teams. Our investigation reveals several key findings: (1) The winning submission in the external tuning ruleset, using DISTRIBUTED SHAMPOO, demonstrates the effectiveness of nondiagonal preconditioning over popular methods like ADAM, even when compared on wall-clock runtime. (2) The winning submission in the self-tuning ruleset, based on the SCHEDULE FREE ADAMW algorithm, demonstrates a new level of effectiveness for completely hyperparameter-free training algorithms. (3) The topscoring submissions were surprisingly robust to workload changes. We also discuss the engineering challenges encountered in ensuring a fair comparison between different training algorithms. These results highlight both the significant progress so far, and the considerable room for further improvements. Submissions. Submitted training algorithms must adhere to the fixed ALGOPERF API (Dahl et al., 2023, Sec. 4.2) and are limited to four submission functions: (1) update_params is responsible for 1 Note, the two speed-ups for SCHEDULE FREE ADAMW are computed across different sets of workloads. 2 github.com/mlcommons/algorithmic-efficiency/[...]/COMPETITION_RULES.md 3 github.com/mlcommons/algorithmic-efficiency/[...]/DOCUMENTATION.md