EMNLP Findings2025
DisastIR: A Comprehensive Information Retrieval Benchmark for Disaster Management
Kai Yin, Xiangjue Dong, Chengkai Liu, Lipai Huang, Yiming Xiao, Zhewei Liu, Ali Mostafavi, James Caverlee
Abstract
Effective disaster management requires timely access to accurate and contextually relevant information. Existing Information Retrieval (IR) benchmarks, however, focus primarily on general or specialized domains, such as medicine or finance, neglecting the unique linguistic complexity and diverse information needs encountered in disaster management scenarios. To bridge this gap, we introduce DisastIR, the first comprehensive IR evaluation benchmark specifically tailored for disaster management. DisastIR comprises 9,600 diverse user queries and more than 1.3 million labeled querypassage pairs, covering 48 distinct retrieval tasks derived from six search intents and eight general disaster categories that include 301 specific event types. Our evaluations of 30 state-ofthe-art retrieval models demonstrate significant performance variances across tasks, with no single model excelling universally. Furthermore, comparative analyses reveal significant performance gaps between general-domain and disaster management-specific tasks, highlighting the necessity of disaster management-specific benchmarks for guiding IR model selection to support effective decision-making in disaster management scenarios. All source codes and DisastIR are available at this repository.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- One Embedder, Any Task: Instruction-Finetuned Text Embeddings cites Task-aware Retrieval with Instructions
- Task-aware Retrieval with Instructions cites One Embedder, Any Task: Instruction-Finetuned Text Embeddings
- Task-aware Retrieval with Instructions cites Promptagator: Few-shot Dense Retrieval From 8 Examples
- DMRetriever: A Family of Models for Improved Text Retrieval in Disaster Management cites Task-aware Retrieval with Instructions
Cited by top-tier papers2
- HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL ConferencesYusuke Sakai, Hidetaka Kamigaito, Taro WatanabeACL 2026 · 18 citations
- DMRetriever: A Family of Models for Improved Text Retrieval in Disaster ManagementKai Yin, Xiangjue Dong, Chengkai Liu, Allen Lin et al.ACL 2026 · 1 citation
Builds on12
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge DistillationJianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo et al.ACL Findings 2024 · 762 citations
- Large Language Models can Accurately Predict Searcher PreferencesPaul Thomas, Seth Spielman, Nick Craswell, Bhaskar MitraSIGIR 2024 · 153 citations
- One Embedder, Any Task: Instruction-Finetuned Text EmbeddingsHongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang et al.ACL Findings 2023 · 134 citations
- Promptagator: Few-shot Dense Retrieval From 8 ExamplesZhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan et al.ICLR 2023 · 46 citations
- Task-aware Retrieval with InstructionsAkari Asai, Timo Schick, Patrick Lewis, Xilun Chen et al.ACL Findings 2023 · 45 citations
Related papers
- DisastQA: A Comprehensive Benchmark for Evaluating Question Answering in Disaster ManagementZhitong Chen, Kai Yin, Xiangjue Dong, Chengkai Liu et al.ACL Findings 2026
- AIR-Bench: Automated Heterogeneous Information Retrieval BenchmarkJianlyu Chen, Nan Wang, Chaofan Li, Bo Wang et al.ACL 2025
- MAIR: A Massive Benchmark for Evaluating Instructed RetrievalWeiwei Sun, Zhengliang Shi, Wu Long, Lingyong Yan et al.EMNLP 2024 · 1 citation
- Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents IntegrationSunhao Dai, Weihao Liu, Yuqi Zhou, Liang Pang et al.ACL Findings 2024 · 12 citations
- CoIR: A Comprehensive Benchmark for Code Information Retrieval ModelsXiangyang Li, Kuicai Dong, Yi Quan Lee, Wei Xia et al.ACL 2025
