Lune

ACL Findings2023

Scaling Laws for BERT in Low-Resource Settings

Gorka Urbizu, Iñaki San Vicente, Xabier Saralegi, Rodrigo Agerri, Aitor Soroa

Abstract

Large language models are very resource intensive, both financially and environmentally, and require an amount of training data which is simply unobtainable for the majority of NLP practitioners. Previous work has researched the scaling laws of such models, but optimal ratios of model parameters, dataset size, and computation costs focused on the large scale. In contrast, we analyze the effect those variables have on the performance of language models in constrained settings, by building three lightweight BERT models (16M/51M/124M parameters) trained over a set of small corpora (5M/25M/125M words). We experiment on four languages of different linguistic characteristics (Basque, Spanish, Swahili and Finnish), and evaluate the models on MLM and several NLU tasks. We conclude that the power laws for parameters, data and compute for lowresource settings differ from the optimal scaling laws previously inferred, and data requirements should be higher. Our insights are consistent across all the languages we study, as well as across the MLM and downstream tasks. Furthermore, we experimentally establish when the cost of using a Transformer-based approach is worth taking, instead of favouring other computationally lighter solutions.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Unified Scaling Laws for Routed Language Models cites Language Models are Few-Shot Learners
  • Training compute-optimal transformer encoder models cites Unsupervised Cross-lingual Representation Learning at Scale
  • Inference Scaling Law for Retrieval Augmented Generation cites Unsupervised Cross-lingual Representation Learning at Scale
  • Inference Scaling Law for Retrieval Augmented Generation cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Training compute-optimal transformer encoder modelsMegi Dervishi, Alexandre Allauzen, Gabriel Synnaeve, Yann LeCunEMNLP 2025 · 1 citation
  2. Inference Scaling Law for Retrieval Augmented GenerationShu Zhou, Yuxuan Ao, Yunyang Xuan, Xin Wang et al.AAAI 2026 · 1 citation
  3. From Retrieval to Translation: Translating Query into Graph-level Clues for Retrieval-Augmented GenerationQichuan Liu, Qinggang Zhang, Yuxuan Hu, Chenfeng Zheng et al.ICML 2026

Builds on8

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
  3. On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong BaselinesMarius Mosbach, Maksym Andriushchenko, Dietrich KlakowICLR 2021 · 448 citations
  4. Unified Scaling Laws for Routed Language ModelsAidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch et al.ICML 2022 · 266 citations
  5. A Systematic Assessment of Syntactic Generalization in Neural Language ModelsJennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox et al.ACL 2020 · 124 citations

Related papers

  1. Not Enough Data to Pre-train Your Language Model? MT to the Rescue!Gorka Urbizu, Iñaki San Vicente, Xabier Saralegi, Ander CorralACL Findings 2023 · 4 citations
  2. From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual TransformersAnne Lauscher, Vinit Ravishankar, Ivan Vulic, Goran GlavasEMNLP 2020 · 235 citations
  3. Language models scale reliably with over-training and on downstream tasksSamir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan et al.ICLR 2025 · 3 citations
  4. OWLS: Scaling Laws for Multilingual Speech Recognition and Translation ModelsWilliam Chen, Jinchuan Tian, Yifan Peng, Brian Yan et al.ICML 2025
  5. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling LawsNikhil Sardana, Jacob P. Portes, Sasha Doubov, Jonathan FrankleICML 2024 · 144 citations
Dusk over the sea between two cliffs drawn in fine vertical lines