ACL Findings2024
Measuring and Addressing Indexical Bias in Information Retrieval
Caleb Ziems, William Barr Held, Jane Dwivedi-Yu, Diyi Yang
Abstract
Information Retrieval (IR) systems are designed to deliver relevant content, but traditional systems may not optimize rankings for fairness, neutrality, or the balance of ideas.Consequently, IR can often introduce indexical biases, or biases in the positional order of documents.Although indexical bias can demonstrably affect people's opinion, voting patterns, and other behaviors, these issues remain understudied as the field lacks reliable metrics and procedures for automatically measuring indexical bias.Towards this end, we introduce the PAIR framework, which supports automatic bias audits for ranked documents or entire IR systems.After introducing DUO, the first general-purpose automatic bias metric, we run an extensive evaluation of 8 IR systems on a new corpus of 32k synthetic and 4.7k natural documents, with 4k queries spanning 1.4k controversial issue topics.A human behavioral study validates our approach, showing that our bias metric can help predict when and how indexical bias will shift a reader's opinion.For data and code, see https://github.com/ SALT-NLP/pair Pos. 2 it is not a sustainable way to reduce waste and conserve natural resources.Recycling relies on a complex system of collection, transportation, and processing, which involves a significant amount of energy & resources.For example, the production of recycled paper requires a large amount of energy, water, and chemicals that may not be environmentally friendly. Pos. 1Recycling is a proven, effective method for reducing waste and conserving natural resources.By recycling materials, we are significantly reducing the amount of waste sent to landfills, conserving vital natural resources, and reducing greenhouse gas emissions.This is because recycling helps to create a circular economy People Contribute to misogyny?Taylor Swift Freud Politics History China gun control sovereignty over Taiwan?sanctions for human rights?right to military weapons?background checks enough?US response appropriate?Cuban Missile Crisis Change international diplomacy?Environment recycling Mandatory for all households?Effective at reducing waste?Applicable outside the west?Positive or negative influence? Has her image been consistent?Cultural appropriation to wear Air Max if not Black?Culture Nike Air Max Are they a symbol of status and wealth?Language Sapir-Whorf Useful for understanding culture?Can it be tested empirically?
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Active Retrieval Augmented Generation cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Cited by top-tier papers2
- BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language ModelsThierry Blankenstein, Jialin Yu, Zixuan Li, Vassilis Plachouras et al.ICLR 2026 · 8 citations
- Detecting Miscitation on the Scholarly Web through LLM-Augmented Text-Rich Graph LearningHuidong Wu, Haojia Xiang, Jingtong Gao, Xiangyu Zhao et al.WWW 2026
Builds on14
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text RetrievalLee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang et al.ICLR 2021 · 1,547 citations
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTOmar Khattab, Matei ZahariaSIGIR 2020 · 1,246 citations
- Active Retrieval Augmented GenerationZhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun et al.EMNLP 2023 · 315 citations
- FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive SummarizationEsin Durmus, He He, Mona T. DiabACL 2020 · 90 citations
Related papers
- An Empirical Study of Position Bias in Modern Information RetrievalZiyang Zeng, Dun Zhang, Jiacheng Li, Panxiang Zou et al.EMNLP Findings 2025 · 1 citation
- Neural Retrievers are Biased Towards LLM-Generated ContentSunhao Dai, Yuqi Zhou, Liang Pang, Weihao Liu et al.KDD 2024 · 26 citations
- Mitigating Source Bias with LLM AlignmentSunhao Dai, Yuqi Zhou, Liang Pang, Zhuoyang Li et al.SIGIR 2025 · 2 citations
- JobFair: A Framework for Benchmarking Gender Hiring Bias in Large Language ModelsZe Wang, Zekun Wu, Xin Guan, Michael Thaler et al.EMNLP Findings 2024 · 14 citations
- Reinforcement Learning to Rank with Pairwise Policy GradientJun Xu, Zeng Wei, Long Xia, Yanyan Lan et al.SIGIR 2020 · 32 citations
