EMNLP Findings2022
What do Large Language Models Learn beyond Language?
Abstract
Large language models (LMs) have rapidly become a mainstay in Natural Language Processing. These models are known to acquire rich linguistic knowledge from training on large amounts of text. In this paper, we investigate if pre-training on text also confers these models with helpful 'inductive biases' for nonlinguistic reasoning. On a set of 19 diverse non-linguistic tasks involving quantitative computations, recognizing regular expressions and reasoning over strings. We find that pretrained models significantly outperform comparable non-pretrained neural models. This remains true also in experiments with training nonpretrained models with fewer parameters to account for model regularization effects. We further explore the effect of text domain on LMs by pretraining models from text from different domains and provenances. Our experiments surprisingly reveal that the positive effects of pre-training persist even when pretraining on multi-lingual text or computer code, and even for text generated from synthetic languages. Our findings suggest a hithertho unexplored deep connection between pre-training and inductive learning abilities of language models 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on8
- CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesZhangyin Feng, Daya Guo, Duyu Tang, Nan Duan et al.EMNLP Findings 2020 · 2,828 citations
- Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTZhiyong Wu, Yun Chen, Ben Kao, Qun LiuACL 2020 · 158 citations
- What they do when in doubt: a study of inductive biases in seq2seq learnersEugene Kharitonov, Rahma ChaabouniICLR 2021 · 29 citations
- Probing Pretrained Language Models for Lexical SemanticsIvan Vulic, Edoardo Maria Ponti, Robert Litschko, Goran Glavas et al.EMNLP 2020 · 26 citations
- Quantifying the Contextualization of Word Representations with Semantic Class ProbingMengjie Zhao, Philipp Dufter, Yadollah Yaghoobzadeh, Hinrich SchützeEMNLP Findings 2020 · 13 citations
Related papers
- LIME: Learning Inductive Bias for Primitives of Mathematical ReasoningYuhuai Wu, Markus N. Rabe, Wenda Li, Jimmy Ba et al.ICML 2021 · 66 citations
- To Code or Not To Code? Exploring Impact of Code in Pre-trainingViraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot et al.ICLR 2025 · 3 citations
- Injecting Numerical Reasoning Skills into Language ModelsMor Geva, Ankit Gupta, Jonathan BerantACL 2020 · 12 citations
- Leveraging the Inductive Bias of Large Language Models for Abstract Textual ReasoningChristopher Michael Rytting, David WingateNeurIPS 2021 · 33 citations
- Classical Planning with LLM-Generated Heuristics: Challenging the State of the Art with Python CodeAugusto B. Corrêa, André Grahl Pereira, Jendrik SeippNeurIPS 2025 · 27 citations
