Lune

EMNLP Findings2025

SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts

Nghiem Thanh Pham, Tung Kieu, Duc-Manh Nguyen, Ha Xuan Son, Nghia Duong-Trung, Danh Le Phuoc

Abstract

Small Language Models (SLMs) offer computational efficiency and accessibility, yet a systematic evaluation of their performance and environmental impact remains lacking. We introduce SLM-Bench, the first benchmark specifically designed to assess SLMs across multiple dimensions, including accuracy, computational efficiency, and sustainability metrics. SLM-Bench evaluates 15 SLMs on 9 NLP tasks using 23 datasets spanning 14 domains. The evaluation is conducted on 4 hardware configurations, providing a rigorous comparison of their effectiveness. Unlike prior benchmarks, SLM-Bench quantifies 11 metrics across correctness, computation, and consumption, enabling a holistic assessment of efficiency tradeoffs. Our evaluation considers controlled hardware conditions, ensuring fair comparisons across models. We develop an open-source benchmarking pipeline with standardized evaluation protocols to facilitate reproducibility and further research. Our findings highlight the diverse trade-offs among SLMs, where some models excel in accuracy while others achieve superior energy efficiency. SLM-Bench sets a new standard for SLM evaluation, bridging the gap between resource efficiency and real-world applicability.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • PIQA: Reasoning about Physical Commonsense in Natural Language cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites TinyBERT: Distilling BERT for Natural Language Understanding
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on16

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
  3. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  4. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  5. LLM-Pruner: On the Structural Pruning of Large Language ModelsXinyin Ma, Gongfan Fang, Xinchao WangNeurIPS 2023 · 994 citations

Related papers

  1. TokenPowerBench: Benchmarking the Power Consumption of LLM InferenceChenxu Niu, Wei Zhang, Jie Li, Yongjian Zhao et al.AAAI 2026 · 12 citations
  2. Unveiling Environmental Impacts of Large Language Model Serving: A Functional Unit ViewYanran Wu, Inez Hua, Yi DingACL 2025 · 15 citations
  3. EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation MetricsDaniil Larionov, Jens Grünwald, Christoph Leiter, Steffen EgerEMNLP Findings 2023 · 4 citations
  4. CMedBench: A Comprehensive Benchmark for Efficient Medical Large Language ModelsShengbo Gao, Jinyang Guo, Lixian Su, Yifu Ding et al.AAAI 2026
  5. Good for the Planet, Bad for Me? Intended and Unintended Consequences of AI Energy Consumption DisclosureMichael Klesel, Uwe MesserCHI 2026 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines