EMNLP Findings2025
SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts
Nghiem Thanh Pham, Tung Kieu, Duc-Manh Nguyen, Ha Xuan Son, Nghia Duong-Trung, Danh Le Phuoc
Abstract
Small Language Models (SLMs) offer computational efficiency and accessibility, yet a systematic evaluation of their performance and environmental impact remains lacking. We introduce SLM-Bench, the first benchmark specifically designed to assess SLMs across multiple dimensions, including accuracy, computational efficiency, and sustainability metrics. SLM-Bench evaluates 15 SLMs on 9 NLP tasks using 23 datasets spanning 14 domains. The evaluation is conducted on 4 hardware configurations, providing a rigorous comparison of their effectiveness. Unlike prior benchmarks, SLM-Bench quantifies 11 metrics across correctness, computation, and consumption, enabling a holistic assessment of efficiency tradeoffs. Our evaluation considers controlled hardware conditions, ensuring fair comparisons across models. We develop an open-source benchmarking pipeline with standardized evaluation protocols to facilitate reproducibility and further research. Our findings highlight the diverse trade-offs among SLMs, where some models excel in accuracy while others achieve superior energy efficiency. SLM-Bench sets a new standard for SLM evaluation, bridging the gap between resource efficiency and real-world applicability.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- PIQA: Reasoning about Physical Commonsense in Natural Language cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- LLM-Pruner: On the Structural Pruning of Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
- LLM-Pruner: On the Structural Pruning of Large Language Models cites TinyBERT: Distilling BERT for Natural Language Understanding
- LLM-Pruner: On the Structural Pruning of Large Language Models cites PIQA: Reasoning about Physical Commonsense in Natural Language
- LLM-Pruner: On the Structural Pruning of Large Language Models cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Builds on16
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
- TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
- LLM-Pruner: On the Structural Pruning of Large Language ModelsXinyin Ma, Gongfan Fang, Xinchao WangNeurIPS 2023 · 994 citations
Related papers
- TokenPowerBench: Benchmarking the Power Consumption of LLM InferenceChenxu Niu, Wei Zhang, Jie Li, Yongjian Zhao et al.AAAI 2026 · 12 citations
- Unveiling Environmental Impacts of Large Language Model Serving: A Functional Unit ViewYanran Wu, Inez Hua, Yi DingACL 2025 · 15 citations
- EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation MetricsDaniil Larionov, Jens Grünwald, Christoph Leiter, Steffen EgerEMNLP Findings 2023 · 4 citations
- CMedBench: A Comprehensive Benchmark for Efficient Medical Large Language ModelsShengbo Gao, Jinyang Guo, Lixian Su, Yifu Ding et al.AAAI 2026
- Good for the Planet, Bad for Me? Intended and Unintended Consequences of AI Energy Consumption DisclosureMichael Klesel, Uwe MesserCHI 2026 · 2 citations
