Lune

EMNLP Findings2023

Energy and Carbon Considerations of Fine-Tuning BERT

Xiaorong Wang, Clara Na, Emma Strubell, Sorelle A. Friedler, Sasha Luccioni

Abstract

Despite the popularity of the pre-train then finetune paradigm in the NLP community, existing work quantifying energy costs and associated carbon emissions has largely focused on language model pre-training. Although a single pre-training run draws substantially more energy than fine-tuning, fine-tuning is performed more frequently by many more individual actors, and thus must be accounted for when considering the energy and carbon footprint of NLP. In order to better characterize the role of finetuning in the landscape of energy and carbon emissions in NLP, we perform a careful empirical study of the computational costs of finetuning across tasks, datasets, hardware infrastructure and measurement modalities. Our experimental results allow us to place fine-tuning energy and carbon costs into perspective with respect to pre-training and inference, and outline recommendations to NLP researchers and practitioners who wish to improve their finetuning energy efficiency.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Hugging Carbon: Quantifying the Training Carbon Emissions of AI Models at Scale cites Energy Considerations of Large Language Model Inference and Efficiency Optimizations
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. An Adaptive Language-Agnostic Pruning Method for Greener Language Models for CodeMootez Saad, José Antonio Hernández López, Boqi Chen, Dániel Varró et al.FSE 2025 · 1 citation
  2. Energy Considerations of Large Language Model Inference and Efficiency OptimizationsJared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk et al.ACL 2025
  3. Hugging Carbon: Quantifying the Training Carbon Emissions of AI Models at ScaleXinlei Wang, Ruibo Ming, Jing Qiu, Junhua Zhao et al.ICML 2026
  4. The Practical Impacts of Theoretical Constructs on Empathy ModelingAllison Lahnala, Charles Welch, David Jurgens, Lucie FlekEMNLP 2025

Builds on2

  1. Scale Efficiently: Insights from Pretraining and Finetuning TransformersYi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus et al.ICLR 2022 · 67 citations
  2. IrEne: Interpretable Energy Prediction for TransformersQingqing Cao, Yash Kumar Lal, Harsh Trivedi, Aruna Balasubramanian et al.ACL 2021

Related papers

  1. Towards Green AI in Fine-tuning Large Language Models via Adaptive BackpropagationKai Huang, Hanyun Yin, Heng Huang, Wei GaoICLR 2024 · 21 citations
  2. A Survey on Model Compression and Acceleration for Pretrained Language ModelsCanwen Xu, Julian J. McAuleyAAAI 2023 · 96 citations
  3. Selecting Informative Contexts Improves Language Model Fine-tuningRichard J. Antonello, Nicole Beckage, Javier Turek, Alexander HuthACL 2021
  4. Distill or Annotate? Cost-Efficient Fine-Tuning of Compact ModelsJunmo Kang, Wei Xu, Alan RitterACL 2023 · 5 citations
  5. NLP Libraries, Energy Consumption and Runtime: An Empirical StudyRajrupa Chattaraj, Sridhar ChimalakondaFSE 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines