Stitchable Neural Networks

Figure 1 . Compared with previous scalable deep learning frameworks. (a) Network compression shrinks a large network into a small one by techniques such as pruning, quantization and knowledge distillation, etc., which is a one-to-one mapping. (b) One-shot neural architecture search first trains a supernet that supports diverse architectural settings and then specializes a subnet given the target resource constraint during deployment, which is a case of one-to-many. (c) Our proposed Stitchable Neural Network directly stitches the off-the-rack family of pretrained models and quickly obtains new networks for efficient model design and deployment in a novel many-to-many paradigm.