Video compression dataset and benchmark of learning-based video-quality metrics

Video-quality measurement is a critical task in video processing. Nowadays, many implementations of new encoding standards -such as AV1, VVC, and LCEVC -use deep-learning-based decoding algorithms with perceptual metrics that serve as optimization objectives. But investigations of the performance of modern video-and image-quality metrics commonly employ videos compressed using older standards, such as AVC. In this paper, we present a new benchmark for video-quality metrics that evaluates video compression. It is based on a new dataset consisting of about 2,500 streams encoded using different standards, including AVC, HEVC, AV1, VP9, and VVC. Subjective scores were collected using crowdsourced pairwise comparisons. The list of evaluated metrics includes recent ones based on machine learning and neural networks. The results demonstrate that new noreference metrics exhibit high correlation with subjective quality and approach the capability of top full-reference metrics. * A. Antsiferova designed methodology and performed results analysis, led the benchmark and the paper preparation, S. Lavrushkin performed results analysis and the paper preparation, M. Smirnov developed the benchmark of no-reference metrics, A. Gushchin developed the benchmark of full-reference metrics, both of them performed the dataset research, results analysis and contributed to the paper preparation, D. Vatolin and D. Kulikov coordinated the benchmark and the dataset creation 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks.