MLSLT: Towards Multilingual Sign Language Translation

Most of the research to date focuses on bilingual sign language translation (BSLT). However, such models are in-efficient in building multilingual sign language translation systems. To solve this problem, we introduce the multilin-gual sign language translation (MSLT) task. It aims to use a single model to complete the translation between multiple sign languages and spoken languages. Then, we propose MLSLT, the first MSLT model, which contains two novel dy-namic routing mechanisms for controlling the degree ofpa-rameter sharing between different languages. Intra-layer language-specific routing controls the proportion of data flowing through shared parameters and language-specific parameters from the token level through a soft gate within the layer, and inter-layer language-specific routing controls and learns the data flow path of different languages at the language level through a soft gate between layers. In order to evaluate the performance of MLSLT, we collect the first publicly available multilingual sign language understanding dataset, Spreadthesign-Ten (SP-10), which contains up to 100 language pairs, e.g., CSL→en, GSG→zh. Experi-mental results show that the average performance of ML-SLT outperforms the baseline MSLT model and the com-bination of multiple BSLT models in many cases. In ad-dition, we also explore zero-shot translation in sign language and find that our model can achieve comparable performance to the supervised BSLT model on some language pairs. Dataset and more details are at https://mlslt.github.io/.