Lune

EMNLP Findings2023

Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization

Ningyu Xu, Qi Zhang, Jingting Ye, Menghan Zhang, Xuanjing Huang

Abstract

Large language models (LLMs) have exhibited considerable cross-lingual generalization abilities, whereby they implicitly transfer knowledge across languages. However, the transfer is not equally successful for all languages, especially for low-resource ones, which poses an ongoing challenge. It is unclear whether we have reached the limits of implicit crosslingual generalization and if explicit knowledge transfer is viable. In this paper, we investigate the potential for explicitly aligning conceptual correspondence between languages to enhance cross-lingual generalization. Using the syntactic aspect of language as a testbed, our analyses of 43 languages reveal a high degree of alignability among the spaces of structural concepts within each language for both encoder-only and decoder-only LLMs. We then propose a meta-learning-based method to learn to align conceptual spaces of different languages, which facilitates zero-shot and fewshot generalization in concept classification and also offers insights into the cross-lingual incontext learning phenomenon. Experiments on syntactic analysis tasks show that our approach achieves competitive results with state-of-theart methods and narrows the performance gap between languages, particularly benefiting those with limited resources.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers cites Multilingual Alignment of Contextual Word Representations
  • Unveiling Linguistic Regions in Large Language Models cites Multilingual Alignment of Contextual Word Representations
  • From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment cites Unveiling Linguistic Regions in Large Language Models
  • From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment cites Exploring Alignment in Shared Cross-lingual Spaces
  • Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
  • Exploring Alignment in Shared Cross-lingual Spaces cites Multilingual Alignment of Contextual Word Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Exploring Alignment in Shared Cross-lingual SpacesBasel Mousi, Nadir Durrani, Fahim Dalvi, Majd Hawasly et al.ACL 2024 · 1 citation
  2. Unveiling Linguistic Regions in Large Language ModelsZhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui et al.ACL 2024
  3. From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons AlignmentChongxuan Huang, Yongshi Ye, Biao Fu, Qifeng Su et al.ACL 2025
  4. Explainability and Interpretability of Multilingual Large Language Models: A SurveyLucas Resck, Isabelle Augenstein, Anna KorhonenEMNLP 2025

Builds on24

  1. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order SensitivityYao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel et al.ACL 2022 · 1,494 citations
  2. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe et al.EMNLP 2022 · 634 citations
  3. From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual TransformersAnne Lauscher, Vinit Ravishankar, Ivan Vulic, Goran GlavasEMNLP 2020 · 235 citations
  4. ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual LearningViet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man et al.EMNLP Findings 2023 · 233 citations
  5. Multilingual Alignment of Contextual Word RepresentationsSteven Cao, Nikita Kitaev, Dan KleinICLR 2020 · 211 citations

Related papers

  1. Meta-XNLG: A Meta-Learning Approach Based on Language Clustering for Zero-Shot Cross-Lingual Transfer and GenerationKaushal Kumar Maurya, Maunendra Sankar DesarkarACL Findings 2022 · 6 citations
  2. Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A ReviewFred Philippy, Siwen Guo, Shohreh HaddadanACL 2023 · 9 citations
  3. Meta-Learning for Fast Cross-Lingual Adaptation in Dependency ParsingAnna Langedijk, Verna Dankers, Phillip Lippe, Sander Bos et al.ACL 2022 · 17 citations
  4. Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMsDanni Liu, Jan NiehuesACL 2025 · 23 citations
  5. Cross-Lingual Pre-Training Based Transfer for Zero-Shot Neural Machine TranslationBaijun Ji, Zhirui Zhang, Xiangyu Duan, Min Zhang et al.AAAI 2020 · 67 citations
Dusk over the sea between two cliffs drawn in fine vertical lines