Towards Better Evaluating Multi-Query Sessions: A Measure Based on the Theory of Planned Behavior

To evaluate multi-query sessions, recent studies usually add a second ''session'' dimension to the query-level evaluation framework, deriving corresponding session-version evaluation metrics such as sDCG, sRBP, and sINST. However, these existing metrics do not sufficiently consider the different impacts of users' expectations of gains and costs on their behaviors such as query reformulation, nor the bounded rationality characteristic of users in expectation management. To address these issues and better explain user behavior in multi-query sessions, we design a user model based on the Theory of Planned Behavior (TPB), which links user expectations to user behaviors. Within the TPB framework, we propose sTPB, a new measure that adapts to users' expectation management modes by considering users' expectations of gains and costs. To demonstrate the effectiveness of sTPB in evaluating multi-query sessions, we compare it with existing session metrics on two publicly available user search behavior datasets. The results show that sTPB significantly outperforms other metrics in terms of both fitting user behavior and measuring user satisfaction. Additionally, we explore the differences between optimal parameters under different user characteristics and task types in session search evaluation. We find that different user characteristics and task types lead to various preferences in users' choices between continuing to examine results and reformulating queries. Our study not only validates the effectiveness of sTPB in evaluating multi-query sessions but also highlights the necessity of considering the influence of user characteristics and task types when designing metrics.