Packed Ensembles for efficient uncertainty estimation

Deep Ensembles (DE) are a prominent approach for achieving excellent performance on key metrics such as accuracy, calibration, uncertainty estimation, and out-of-distribution detection. However, hardware limitations of real-world systems constrain users to smaller ensembles and lower-capacity networks, significantly deteriorating their performance. We introduce Packed-Ensembles (PE), a strategy to design and train lightweight structured ensembles by carefully modulating the dimension of their encoding space. We leverage grouped convolutions to parallelize the ensemble into a single shared backbone and forward pass to reduce the number of parameters and improve training and inference speeds when using mixed precision. PE is designed to operate within the memory limits of a standard neural network. Our extensive research indicates that PE accurately preserves the properties of DE, such as diversity, and performs equally well in terms of accuracy, calibration, out-of-distribution detection, and robustness to distribution shift. We make our code available at github.com/ENSTA-U2IS-AI/torch-uncertainty. 2 3 4 5 6 Images/sec (×10 3 ) 78 79 80 81 82 Accuracy (%) 10M 20M 90M Packed-Ensembles (2, 4, 1) Packed-Ensembles (2, 4, 2) Deep Ensembles (×4) Single MIMO (4)