EfficientViT: Lightweight Multi-Scale Attention for High-Resolution Dense Prediction

High-resolution dense prediction enables many appealing real-world applications, such as computational photography, autonomous driving, etc. However, the vast computational cost makes deploying state-of-the-art highresolution dense prediction models on hardware devices difficult. This work presents EfficientViT, a new family of high-resolution vision models with novel lightweight multiscale attention. Unlike prior high-resolution dense prediction models that rely on heavy self-attention, hardwareinefficient large-kernel convolution, or complicated topology structure to obtain good performances, our lightweight multi-scale attention achieves a global receptive field and multi-scale learning (two critical features for highresolution dense prediction) with only lightweight and hardware-efficient operations. As such, EfficientViT delivers remarkable performance gains over previous state-ofthe-art high-resolution dense prediction models with significant speedup on diverse hardware platforms, including mobile CPU, edge GPU, and cloud GPU. Without performance loss on Cityscapes, our EfficientViT provides up to 8.8× and 3.8× GPU latency reduction over SegFormer and SegNeXt, respectively. For super-resolution, EfficientViT provides up to 6.4× speedup over Restormer while providing 0.11dB gain in PSNR.

SegFormer Untitled 1 Untitled 2 25 80.5 50.5 79.8 Untitled 3 74 82.1 124.6 81.3 243.7 78.5 Untitled 4 179 83.0 275.7 82.6 717.1 81.0