Lune

ACL Findings2023

Transitioning from benchmarks to a real-world case of information-seeking in Scientific Publications

Chyrine Tahri, Aurore Bochnakian, Patrick Haouat, Xavier Tannier

摘要

Although recent years have been marked by incredible advances in the whole development process of NLP systems, there are still blind spots in characterizing what is still hampering real-world adoption of models in knowledgeintensive settings. In this paper, we illustrate through a real-world zero-shot text search case for information seeking in scientific papers, the masked phenomena that the current process of measuring performance might not reflect, even when benchmarks are, in appearance, faithfully representative of the task at hand. In addition to experimenting with TREC-COVID and NFCorpus, we provide an industrial, expertcarried/annotated, case of studying vitamin B's impact on health. We thus discuss the misalignment between solely focusing on single-metric performance as a criterion for model choice and relevancy as a subjective measure for meeting a user's need.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • SciRepEval: A Multi-Format Benchmark for Scientific Document Representations 引用了 SPECTER: Document-level Representation Learning using Citation-informed Transformers
  • SciRepEval: A Multi-Format Benchmark for Scientific Document Representations 引用了 Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings
  • Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings 引用了 SPECTER: Document-level Representation Learning using Citation-informed Transformers
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper4

  1. Document Ranking with a Pretrained Sequence-to-Sequence ModelRodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, Jimmy LinEMNLP Findings 2020 · 469 次引用
  2. Neighborhood Contrastive Learning for Scientific Document Representations with Citation EmbeddingsMalte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp 等EMNLP 2022 · 46 次引用
  3. SciRepEval: A Multi-Format Benchmark for Scientific Document RepresentationsAmanpreet Singh, Mike D'Arcy, Arman Cohan, Doug Downey 等EMNLP 2023 · 45 次引用
  4. SPECTER: Document-level Representation Learning using Citation-informed TransformersArman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey 等ACL 2020 · 20 次引用

相关 Paper

  1. Examining the robustness of LLM evaluation to the distributional assumptions of benchmarksCharlotte Siska, Katerina Marazopoulou, Melissa Ailem, James BonoACL 2024
  2. ResearchArena: Benchmarking Large Language Models' Ability to Collect and Organize Information as Research AgentsHao Kang, Chenyan XiongEMNLP Findings 2025 · 3 次引用
  3. The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT InteractionsSiru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong 等EMNLP 2023 · 10 次引用
  4. Goal Driven Discovery of Distributional Differences via Language DescriptionsRuiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn 等NeurIPS 2023 · 81 次引用
  5. Not What the Doctor Ordered: Surveying LLM-based De-identification and Quantifying Clinical Information LossKiana Aghakasiri, Noopur Zambare, JoAnn Thai, Carrie Ye 等EMNLP 2025 · 1 次引用
黄昏的海面,两侧是细线勾勒的悬崖