Lune

EMNLP Findings2024

Representational Isomorphism and Alignment of Multilingual Large Language Models

Di Wu, Yibin Lei, Andrew Yates, Christof Monz

Abstract

In this paper, we investigate the capability of Large Language Models (LLMs) to represent texts in multilingual contexts. Our findings show that sentence representations derived from LLMs exhibit a high degree of isomorphism across languages. This existing isomorphism can facilitate representational alignments in zero-shot and few-shot settings. Specifically, by applying a contrastive objective at the representation level with only a few translation pairs (e.g., 100), we substantially improve models' performance on Semantic Textual Similarity (STS) tasks across languages. This representation-level approach proves to be more efficient and effective for semantic alignment than continued pretraining or instruction tuning. Interestingly, we also observe substantial STS improvements within individual languages, even without a monolingual objective specifically designed for this purpose. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
  • Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
  • PromptBERT: Improving BERT Sentence Embeddings with Prompts cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
  • SimCSE: Simple Contrastive Learning of Sentence Embeddings cites A Simple Framework for Contrastive Learning of Visual Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMsDanni Liu, Jan NiehuesACL 2025 · 23 citations
  2. Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language ModelsAnirudh Sundar, Sinead Williamson, Katherine Metcalf, Barry-John Theobald et al.ACL 2025 · 8 citations
  3. Understanding Subword Compositionality of Large Language ModelsQiwei Peng, Yekun Chai, Anders SøgaardEMNLP 2025

Builds on12

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
  4. A Survey on In-context LearningQingxiu Dong, Lei Li, Damai Dai, Ce Zheng et al.EMNLP 2024 · 479 citations
  5. PromptBERT: Improving BERT Sentence Embeddings with PromptsTing Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang et al.EMNLP 2022 · 148 citations

Related papers

  1. AlignX: Advancing Multilingual Large Language Models with Multilingual Representation AlignmentMengyu Bu, Shaolei Zhang, Zhongjun He, Hua Wu et al.EMNLP 2025
  2. Effective Fine-Tuning Methods for Cross-lingual AdaptationTao Yu, Shafiq R. JotyEMNLP 2021 · 7 citations
  3. Getting More from Less: Large Language Models are Good Spontaneous Multilingual LearnersShimao Zhang, Changjiang Gao, Wenhao Zhu, Jiajun Chen et al.EMNLP 2024 · 1 citation
  4. Pruning Multilingual Large Language Models for Multilingual InferenceHwichan Kim, Jun Suzuki, Tosho Hirasawa, Mamoru KomachiEMNLP Findings 2024 · 1 citation
  5. Probing the Emergence of Cross-lingual Alignment during LLM TrainingHetong Wang, Pasquale Minervini, Edoardo M. PontiACL Findings 2024 · 13 citations
Dusk over the sea between two cliffs drawn in fine vertical lines