GSPO: A Graph Substitution and Parallelization Joint Optimization Framework for DNN Inference

This work proposes GSPO, an automatic unified framework that jointly applies graph substitution and parallelization for DNN inference. GSPO uses a joint optimization computation graph (JOCG) to represent graph substitution and parallelization at the operator level. Then, a novel cost model customized for joint optimization is used to evaluate the computation graph execution time quickly. With the graph partition and backtracking search algorithm, GSPO can find the optimal joint optimization solution within an acceptable search time. Compared to existing frameworks applying graph substitution or parallelization, GSPO can achieve up to 27.1% end-to-end performance improvement and reduce search time by up to 94.3%.