Lune

ACL Findings2023

Second Language Acquisition of Neural Language Models

Miyu Oba, Tatsuki Kuribayashi, Hiroki Ouchi, Taro Watanabe

Abstract

With the success of neural language models (LMs), their language acquisition has gained much attention. This work sheds light on the second language (L2) acquisition of LMs, while previous work has typically explored their first language (L1) acquisition. Specifically, we trained bilingual LMs with a scenario similar to human L2 acquisition and analyzed their cross-lingual transfer from linguistic perspectives. Our exploratory experiments demonstrated that the L1 pretraining accelerated their linguistic generalization in L2, and language transfer configurations (e.g., the L1 choice, and presence of parallel texts) substantially affected their generalizations. These clarify their (non-)human-like L2 acquisition in particular aspects. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Modeling Nonnative Sentence Processing with L2 Language Models cites Unsupervised Cross-lingual Representation Learning at Scale
  • Modeling Nonnative Sentence Processing with L2 Language Models cites Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
  • Language models are multilingual chain-of-thought reasoners cites Unsupervised Cross-lingual Representation Learning at Scale
  • Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models cites Unsupervised Cross-lingual Representation Learning at Scale
  • Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models cites Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Modeling Nonnative Sentence Processing with L2 Language ModelsTatsuya Aoyama, Nathan SchneiderEMNLP 2024 · 1 citation
  2. Can LLMs Simulate L2-English Dialogue? An Information-Theoretic Analysis of L1-Dependent BiasesRena Wei Gao, Xuetong Wu, Tatsuki Kuribayashi, Mingrui Ye et al.ACL 2025

Builds on10

  1. Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
  2. Masked Language Model ScoringJulian Salazar, Davis Liang, Toan Q. Nguyen, Katrin KirchhoffACL 2020 · 167 citations
  3. Language models are multilingual chain-of-thought reasonersFreda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang et al.ICLR 2023 · 52 citations
  4. Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language ModelsIsabel Papadimitriou, Dan JurafskyEMNLP 2020 · 40 citations
  5. Pretraining with Artificial Language: Studying Transferable Knowledge in Language ModelsRyokan Ri, Yoshimasa TsuruokaACL 2022 · 40 citations

Related papers

  1. Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language ModelsTerra Blevins, Hila Gonen, Luke ZettlemoyerEMNLP 2022 · 13 citations
  2. Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?Tannon Kew, Florian Schottmann, Rico SennrichEMNLP Findings 2024 · 11 citations
  3. Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language StructureYuan Chai, Yaobo Liang, Nan DuanACL 2022
  4. Analysis of Multi-Source Language Training in Cross-Lingual TransferSeong Hoon Lim, Taejun Yun, Jinhyeon Kim, Jihun Choi et al.ACL 2024 · 1 citation
  5. Multi-Lingual Acquisition on Multimodal Pre-training for Cross-modal RetrievalLiang Zhang, Anwen Hu, Qin JinNeurIPS 2022 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines