ACL Findings2023
Transitioning from benchmarks to a real-world case of information-seeking in Scientific Publications
Chyrine Tahri, Aurore Bochnakian, Patrick Haouat, Xavier Tannier
Abstract
Although recent years have been marked by incredible advances in the whole development process of NLP systems, there are still blind spots in characterizing what is still hampering real-world adoption of models in knowledgeintensive settings. In this paper, we illustrate through a real-world zero-shot text search case for information seeking in scientific papers, the masked phenomena that the current process of measuring performance might not reflect, even when benchmarks are, in appearance, faithfully representative of the task at hand. In addition to experimenting with TREC-COVID and NFCorpus, we provide an industrial, expertcarried/annotated, case of studying vitamin B's impact on health. We thus discuss the misalignment between solely focusing on single-metric performance as a criterion for model choice and relevancy as a subjective measure for meeting a user's need.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- SciRepEval: A Multi-Format Benchmark for Scientific Document Representations cites SPECTER: Document-level Representation Learning using Citation-informed Transformers
- SciRepEval: A Multi-Format Benchmark for Scientific Document Representations cites Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings
- Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings cites SPECTER: Document-level Representation Learning using Citation-informed Transformers
Builds on4
- Document Ranking with a Pretrained Sequence-to-Sequence ModelRodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, Jimmy LinEMNLP Findings 2020 · 469 citations
- Neighborhood Contrastive Learning for Scientific Document Representations with Citation EmbeddingsMalte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp et al.EMNLP 2022 · 46 citations
- SciRepEval: A Multi-Format Benchmark for Scientific Document RepresentationsAmanpreet Singh, Mike D'Arcy, Arman Cohan, Doug Downey et al.EMNLP 2023 · 45 citations
- SPECTER: Document-level Representation Learning using Citation-informed TransformersArman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey et al.ACL 2020 · 20 citations
Related papers
- Examining the robustness of LLM evaluation to the distributional assumptions of benchmarksCharlotte Siska, Katerina Marazopoulou, Melissa Ailem, James BonoACL 2024
- ResearchArena: Benchmarking Large Language Models' Ability to Collect and Organize Information as Research AgentsHao Kang, Chenyan XiongEMNLP Findings 2025 · 3 citations
- The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT InteractionsSiru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong et al.EMNLP 2023 · 10 citations
- Goal Driven Discovery of Distributional Differences via Language DescriptionsRuiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn et al.NeurIPS 2023 · 81 citations
- Not What the Doctor Ordered: Surveying LLM-based De-identification and Quantifying Clinical Information LossKiana Aghakasiri, Noopur Zambare, JoAnn Thai, Carrie Ye et al.EMNLP 2025 · 1 citation
