Bridging the Lab and the Wild: Behavioral Experiments as a Pathway to QoE Research Closer to Realistic Environment

A high effort in Quality of Experience (QoE) research has been put into subjective assessment to determine the perceived quality of video. Most laboratory experiments follow guidelines from the ITU-T Recommendations, which suggest Absolute Category Rating (ACR) as a method to conduct such experiments. However, this method of video assessment radically limits confounding variables, is unnatural, and is far from how people cope with quality degradation and the cues they receive in these situations. This paper addresses this issue and proposes a more realistic subjective experiment based on the participant's behavior. Instead of passively rating the degraded quality of silent videos, we created the possibility to react to the annoying quality of chosen Netflix movies and reward participants by increasing the quality to the best possible. To cope with the data obtained, we adapted the method of fitting psychometric functions known in neuroscience, auditory science, animal science, and psychology. As a result, we obtained a more comprehensive image of the participants and their perceived quality, including their consistency, lapses, and differences. We estimated the parameters using Maximum Likelihood Estimation (MLE) and evaluated the goodness-of-fit of three S-shaped functions: Weibull, cumulative normal, and logistic. In the end, as the most common, we analyze the basic properties of the fitted functions, such as the Point of Subjective Equality (PSE), confidence intervals, and slope (β). We examine their role in describing individual differences among 34 subjects.