Boosting Image Quality Assessment Through Efficient Transformer Adaptation with Local Feature Enhancement

Image Quality Assessment (IQA) constitutes a funda-mental task within the field of computer vision, yet it re-mains an unresolved challenge, owing to the intricate dis-tortion conditions, diverse image contents, and limited availability of data. Recently, the community has wit-nessed the emergence of numerous large-scale pretrained foundation models. However, it remains an open problem whether the scaling law in high-level tasks is also appli-cable to IQA tasks which are closely related to low-level clues. In this paper, we demonstrate that with a proper in-jection of local distortion features, a larger pretrained vision transformer (ViT) foundation model performs better in IQA tasks. Specifically, for the lack of local distortion structure and inductive bias of the large-scale pretrained ViT, we use another pretrained convolution neural networks (CNNs), which is well known for capturing the local structure, to extract multi-scale image features. Further, we propose a local distortion extractor to obtain local distortion features from the pretrained CNNs and a local distortion in-jector to inject the local distortion features into ViT. By only training the extractor and injector, our method can benefit from the rich knowledge in the powerful foundation models and achieve state-of-the-art performance on popular IQA datasets, indicating that IQA is not only a low-level problem but also benefits from stronger high-level features drawn from large-scale pretrained models. Codes are publicly available at: https://github.com/NeosXu/LoDa.