Lune

EMNLP Findings2023

Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distribution

Jaap Jumelet, Willem H. Zuidema

Abstract

We present a setup for training, evaluating and interpreting neural language models, that uses artificial, language-like data. The data is generated using a massive probabilistic grammar (based on state-split PCFGs), that is itself derived from a large natural language corpus, but also provides us complete control over the generative process. We describe and release both grammar and corpus, and test for the naturalness of our generated data. This approach allows us to define closed-form expressions to efficiently compute exact lower bounds on obtainable perplexity using both causal and masked language modelling. Our results show striking differences between neural language modelling architectures and training objectives in how closely they allow approximating the lower bound on perplexity. Our approach also allows us to directly compare learned representations to symbolic rules in the underlying source. We experiment with various techniques for interpreting model behaviour and learning dynamics. With access to the underlying true source, our results show striking differences and outcomes in learning dynamics between different classes of words. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Feature Interactions Reveal Linguistic Structure in Language Models cites "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. On the In-context Generation of Language ModelsZhongtao Jiang, Yuanzhe Zhang, Kun Luo, Xiaowei Yuan et al.EMNLP 2024
  2. Test-Time Learning for Large Language ModelsJinwu Hu, Zitian Zhang, Guohao Chen, Xutao Wen et al.ICML 2025
  3. Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning PerspectiveBishwamittra Ghosh, Soumi Das, Till Speicher, Qinyuan Wu et al.ACL 2026
  4. What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular LanguagesNadav Borenstein, Anej Svete, Robin Chan, Josef Valvoda et al.ACL 2024

Builds on11

  1. Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
  2. An empirical analysis of compute-optimal large language model trainingJordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya et al.NeurIPS 2022 · 566 citations
  3. Masked Language Model ScoringJulian Salazar, Davis Liang, Toan Q. Nguyen, Katrin KirchhoffACL 2020 · 167 citations
  4. "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text ClassificationJasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm et al.EMNLP 2022 · 29 citations
  5. Feature Interactions Reveal Linguistic Structure in Language ModelsJaap Jumelet, Willem H. ZuidemaACL Findings 2023 · 7 citations

Related papers

  1. Evaluating Distributional Distortion in Neural Language ModelingBenjamin LeBrun, Alessandro Sordoni, Timothy J. O'DonnellICLR 2022 · 26 citations
  2. Towards a theory of how the structure of language is acquired by deep neural networksFrancesco Cagnetta, Matthieu WyartNeurIPS 2024 · 33 citations
  3. Examining the Inductive Bias of Neural Language Models with Artificial LanguagesJennifer C. White, Ryan CotterellACL 2021
  4. Deep networks learn to parse uniform-depth context-free languages from local statisticsJack T. Parley, Francesco Cagnetta, Matthieu WyartICML 2026 · 4 citations
  5. Unnatural language processing: How do language models handle machine-generated prompts?Corentin Kervadec, Francesca Franzon, Marco BaroniEMNLP Findings 2023 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines