Cross-Lingual Generalization and Compression: From Language-Specific to Shared Neurons

Multilingual language models (MLLMs) have demonstrated remarkable abilities to transfer knowledge across languages, despite being trained without explicit cross-lingual supervision. We analyze the parameter spaces of three MLLMs to study how their representations evolve during pre-training, observing patterns consistent with compression: models initially form language-specific representations, which gradually converge into cross-lingual abstractions as training progresses. Through probing experiments, we observe a clear transition from uniform language identification capabilities across layers to more specialized layer functions. For deeper analysis, we focus on neurons that encode distinct semantic concepts. By tracing their development during pre-training, we show how they gradually align across languages. Notably, we identify specific neurons that emerge as increasingly reliable predictors for the same concepts across languages. This alignment manifests concretely in generation: once an MLLM exhibits cross-lingual generalization according to our measures, we can select concept-specific neurons identified from, e.g., Spanish text and manipulate them to guide token predictions. Remarkably, rather than generating Spanish text, the model produces semantically coherent English text. This demonstrates that cross-lingually aligned neurons encode generalized semantic representations, independent of the original language encoding. 2 Two checkpoints each from BLOOM-560M and BLOOM-7B1 were excluded from our analysis, as they appear to be corrupted (cf. https://huggingface.co/ bigscience/bloom-560m-intermediate/discussions/2 and https://huggingface.co/bigscience/ bloom-7b1-intermediate/discussions ). (a) Early training stage (step 1000). (b) Late training stage (step 400 000).