(Everything You Never Knew You Needed To Know About) Rank-Biased Measurement For Web Search

Measurement is a fundamental human activity. It is through measurement that we identify opportunity, and via opportunity that we seek improvement. In information retrieval and web search we measure how good search engines are at ordering answers to user queries, how close two ranked lists are to each other, how good LLMs are at re-ranking sets of candidate documents, and how close generated answer sentences are to the ideal output. This talk begins by motivating the well-known ''top-weighted'' measurements that are used when assessing the quality of a ranked list of documents against known annotations (qrels). In particular, Moffat and Zobel [2] suggest use of a parameterized geometric distribution to assign weights to ranks, arguing that a distribution with a bounded infinite sum is preferable to ones with unbounded sums; and also motivating the geometric sequence via a proposed user browsing model, the first time this had been done. Their rankbiased precision mechanism has the further advantage of allowing the extent of the uncertainty to be quantified, where uncertainty arises whenever the qrels are not comprehensive.