Identifying Open Challenges in Language Identification

Automatic language identification is a core problem of many Natural Language Processing (NLP) pipelines. A wide variety of architectures and benchmarks have been proposed with often near-perfect performance. Although previous studies have focused on certain challenging setups (i.e. cross-domain, short inputs), a systematic comparison is missing. We propose a benchmark that allows us to test for the effect of input size, training data size, domain, number of languages, scripts, and language families on performance. We evaluate five popular models on this benchmark and identify which open challenges remain for this task as well as which architectures achieve robust performance. We find that cross-domain setups are the most challenging (although arguably most relevant), and that number of languages, variety in scripts, and variety in language families have only a small impact on performance. We also contribute practical takeaways: training with 1,000 instances per language and a maximum input length of 100 characters is enough for robust language identification. Based on our findings, we train an accurate (94.41%) multidomain language identification model on 2,034 languages, for which we also provide an analysis of the remaining errors. 1References