EMNLP Findings2023
Energy and Carbon Considerations of Fine-Tuning BERT
Xiaorong Wang, Clara Na, Emma Strubell, Sorelle A. Friedler, Sasha Luccioni
Abstract
Despite the popularity of the pre-train then finetune paradigm in the NLP community, existing work quantifying energy costs and associated carbon emissions has largely focused on language model pre-training. Although a single pre-training run draws substantially more energy than fine-tuning, fine-tuning is performed more frequently by many more individual actors, and thus must be accounted for when considering the energy and carbon footprint of NLP. In order to better characterize the role of finetuning in the landscape of energy and carbon emissions in NLP, we perform a careful empirical study of the computational costs of finetuning across tasks, datasets, hardware infrastructure and measurement modalities. Our experimental results allow us to place fine-tuning energy and carbon costs into perspective with respect to pre-training and inference, and outline recommendations to NLP researchers and practitioners who wish to improve their finetuning energy efficiency.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers4
- An Adaptive Language-Agnostic Pruning Method for Greener Language Models for CodeMootez Saad, José Antonio Hernández López, Boqi Chen, Dániel Varró et al.FSE 2025 · 1 citation
- Energy Considerations of Large Language Model Inference and Efficiency OptimizationsJared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk et al.ACL 2025
- Hugging Carbon: Quantifying the Training Carbon Emissions of AI Models at ScaleXinlei Wang, Ruibo Ming, Jing Qiu, Junhua Zhao et al.ICML 2026
- The Practical Impacts of Theoretical Constructs on Empathy ModelingAllison Lahnala, Charles Welch, David Jurgens, Lucie FlekEMNLP 2025
Builds on2
Related papers
- Towards Green AI in Fine-tuning Large Language Models via Adaptive BackpropagationKai Huang, Hanyun Yin, Heng Huang, Wei GaoICLR 2024 · 21 citations
- A Survey on Model Compression and Acceleration for Pretrained Language ModelsCanwen Xu, Julian J. McAuleyAAAI 2023 · 96 citations
- Selecting Informative Contexts Improves Language Model Fine-tuningRichard J. Antonello, Nicole Beckage, Javier Turek, Alexander HuthACL 2021
- Distill or Annotate? Cost-Efficient Fine-Tuning of Compact ModelsJunmo Kang, Wei Xu, Alan RitterACL 2023 · 5 citations
- NLP Libraries, Energy Consumption and Runtime: An Empirical StudyRajrupa Chattaraj, Sridhar ChimalakondaFSE 2025 · 1 citation
