Representational Isomorphism and Alignment of Multilingual Large Language Models

In this paper, we investigate the capability of Large Language Models (LLMs) to represent texts in multilingual contexts. Our findings show that sentence representations derived from LLMs exhibit a high degree of isomorphism across languages. This existing isomorphism can facilitate representational alignments in zero-shot and few-shot settings. Specifically, by applying a contrastive objective at the representation level with only a few translation pairs (e.g., 100), we substantially improve models' performance on Semantic Textual Similarity (STS) tasks across languages. This representation-level approach proves to be more efficient and effective for semantic alignment than continued pretraining or instruction tuning. Interestingly, we also observe substantial STS improvements within individual languages, even without a monolingual objective specifically designed for this purpose. 1