Channel Regeneration: Improving Channel Utilization for Compact DNNs

Overparameterized deep neural networks have redundant neurons that do not contribute to the network's accuracy. In this paper, we introduce a novel channel regeneration technique that reinvigorates these redundant channels of efficient architectures by re-initializing its batch normalization scaling factor γ. This re-initialization of BN γ of these channels promotes regular weight updates during training. Furthermore, we show that channel regeneration encourages the channels to contribute equally to the learned representation and further boosts the generalization accuracy. We apply our technique at regular intervals of the training cycle to improve channel utilization. The solutions proposed in previous works either raise the total computational cost or increase the model complexity. Integrating the channel regeneration technique into the training methodology of efficient architectures requires minimal effort and comes at no additional cost in size or memory. Extensive experiments on several image classification benchmarks and on semantic segmentation task demonstrate the effectiveness of applying the channel regeneration technique to compact architectures.