RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs

Minimal pairs are a well-established approach to evaluating the grammatical knowledge of language models. However, existing resources for minimal pairs address a limited number of languages and lack diversity of language-specific grammatical phenomena. This paper introduces the Russian Benchmark of Linguistic Minimal Pairs (RuBLiMP), which includes 45k pairs of sentences that differ in grammaticality and isolate a morphological, syntactic, or semantic phenomenon. In contrast to existing benchmarks of linguistic minimal pairs, RuBLiMP is created by applying linguistic perturbations to automatically annotated sentences from open text corpora and decontaminating test data. We describe the data collection protocol and present the results of evaluating 25 language models in various scenarios. We find that the widely used LMs for Russian are sensitive to morphological and agreement-oriented contrasts, but fall behind humans on phenomena requiring the understanding of structural relations, negation, transitivity, and tense. RuBLiMP, the codebase, and other materials are publicly available. * Equal contribution. † Work is partially done while at HSE University. Parse Sentence Extraction (a) root spal 'sleep' Vpervye 'For the first time' kosmonavt 'an astronaut' nevesomosti 'zero gravity' v 'in' obl case nsubj advmod Sentence Annotation Minimal Pair Generation adposition government Vpervye kosmonavt spal v nevesomosti. Vpervye kosmonavt spal v nevesomost'. subject-predicate agreement (number) Vpervye kosmonavt spal v nevesomosti. Vpervye kosmonavt spali v nevesomosti. subject-predicate agreement (gender) Vpervye kosmonavt spal v nevesomosti. Vpervye kosmonavt spalo v nevesomosti.