Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs

While large language models demonstrate remarkable capabilities at task-specific applications through fine-tuning, extending these benefits across diverse languages is essential for broad accessibility. However, effective crosslingual transfer is hindered by LLM performance gaps across languages and the scarcity of fine-tuning data in many languages. Through analysis of LLM internal representations from over 1,000+ language pairs, we discover that middle layers exhibit the strongest potential for cross-lingual alignment. Building on this finding, we propose a middle-layer alignment objective integrated into task-specific training. Our experiments on slot filling, machine translation, and structured text generation show consistent improvements in cross-lingual transfer, especially to lower-resource languages. The method is robust to the choice of alignment languages and generalizes to languages unseen during alignment. Furthermore, we show that separately trained alignment modules can be merged with existing task-specific modules, improving cross-lingual capabilities without full re-training. Our code is publicly available 1 . 0 4 8 12 16 20 24 28 32 Layer ID 0 50 100 Avg. retrieval accuracy (%) Llama 3 0 4 8 12 16 20 24 28 Layer ID Qwen 2.5 Overall Low-res. (a) Cross-lingual semantic alignment (measured by average retrieval accuracy over 35 languages and 1190 language directions) varies by layer, with the middle layer showing the highest score. Lower-resource languages are poorly aligned. 0 20 40 60 0 25 50 75 100 Transfer result Llama 3 Correlation: 0.56 F1 0 20 40 60 Qwen 2.5 Correlation: 0.70 Cross-lingual representation retrieval accuracy (%) (b) Positive correlation between base model cross-lingual semantic alignment and downstream transfer performance.