ACL Findings2026
LLMs as Lab Engineers: A Benchmark for Analytical Method Lifecycle Management
Xiaoyi Chen, Mahsa Monshizadeh, Chaoqi Zhang, Jianjun Lang, Yang Wu, Genevieve Mortensen, Xiaozhong Liu, Haixu Tang
Abstract
We introduce ChemBench, a comprehensive benchmark for evaluating LLMs' capabilities in analytical chemistry scenarios.Unlike existing benchmarks focused on factual knowledge, ChemBench assesses model abilities to provide contextualized, practical guidance for complex analytical chemistry challenges, including instrument readiness checks, system suitability testing, method development, and troubleshooting for both liquid chromatography coupled mass spectrometry (LC-MS) and Gas Chromatography-Mass Spectrometry (GC-MS) platforms.We evaluate three enhancement approaches: chemistryspecialized models, human-guided Chain-of-Thought reasoning, and Retrieval-Augmented Generation (RAG).Our findings reveal that general-purpose commercial models often outperform domain-specialized ones, while RAG and reasoning significantly improve performance.The six-dimension evaluation framework (specificity, correctness, usefulness, feasibility, misinformation risk, and error handling) provides valuable insights into LLMs' real-world utility for chemistry researchers, establishing a foundation for developing more effective AI assistants for scientific research.1 Benchmark
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on5
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Document Ranking with a Pretrained Sequence-to-Sequence ModelRodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, Jimmy LinEMNLP Findings 2020 · 469 citations
- RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective AugmentationFangyuan Xu, Weijia Shi, Eunsol ChoiICLR 2024 · 260 citations
- Precise Zero-Shot Dense Retrieval without Relevance LabelsLuyu Gao, Xueguang Ma, Jimmy Lin, Jamie CallanACL 2023 · 211 citations
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific ResearchLiangtai Sun, Yang Han, Zihan Zhao, Da Ma et al.AAAI 2024 · 150 citations
Related papers
- MetaBench: A Multi-task Benchmark for Assessing LLMs in MetabolomicsYuxing Lu, Xukai Zhao, J. Ben Tamo, Micky C. Nnamdi et al.ACL 2026 · 1 citation
- CheMM-R1: Enhancing Chemical Structure Recognition and Elucidation with Reasoning Multimodal Large Language ModelsLiting Huang, Zhihao Zhang, Shoujin WangACL Findings 2026
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language ModelsXiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu et al.ICML 2024 · 220 citations
- MT-Mol: Multi Agent System with Tool-based Reasoning for Molecular OptimizationHyomin Kim, Yunhui Jang, Sungsoo AhnEMNLP Findings 2025 · 6 citations
- RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference AlignmentZhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao et al.ACL Findings 2025 · 1 citation
