Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?

The adaption of multilingual pre-trained Large Language Models (LLMs) into eloquent and helpful assistants is essential to facilitate their use across different language regions. In that spirit, we are the first to conduct an extensive study of the performance of multilingual models instruction-tuned on different language compositions on parallel instruction-tuning benchmarks across a selection of the most spoken Indo-European languages. We systematically examine the effects of language and instruction dataset size on a mid-sized and a large, multilingual LLMs by instruction-tuning them on parallel instruction-tuning datasets. Our results demonstrate that instruction-tuning on parallel instead of monolingual corpora benefits crosslingual instruction following capabilities by up to 9.9%. Furthermore, we show that the Superficial Alignment Hypothesis does not hold in general, as the investigated multilingual 7B parameter model presents a counter-example requiring large-scale instruction-tuning datasets. Finally, we conduct a human annotation study to understand the alignment between humanbased and GPT-4-based evaluation within multilingual chat scenarios. Request (reasoning) -Turn 1 David has three sisters. Each of them has one brother. How many brothers does David have? Reference Solution David has no brother. He is the one brother of his three sisters. Bactrian-DE David has two brothers. Judgment The assistant's answer is incorrect. [. . . ] Rating: [[2]] Bactrian-DEFRITES David has six brothers. Judgment The assistant's answer is incorrect. [. . . ] Rating: [[1]] Bactrian-DEFRITES-sampled David has two brothers. Judgment The assistant's answer is incorrect. [. . . ] Rating: [[2]] Bactrian-ENDEFRITES David has one brother. Judgment The assistant's answer is correct.