Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly

Building on Petroni et al. ( 2019 ), we propose two new probing tasks analyzing factual knowledge stored in Pretrained Language Models (PLMs). ( 1 ) Negation. We find that PLMs do not distinguish between negated ("Birds cannot [MASK]") and non-negated ("Birds can [MASK]") cloze questions. (2) Mispriming. Inspired by priming methods in human psychology, we add "misprimes" to cloze questions ("Talk? Birds can [MASK]"). We find that PLMs are easily distracted by misprimes. These results suggest that PLMs still have a long way to go to adequately learn human-like factual knowledge. cloze question true top 3 words generated with log probs Google RE O Marcel Oopa died in the city of [MASK]. Paris Paris (-2.3), Lausanne (-3.3), Brussels (-3.3) N Marcel Oopa did not die in the city of [MASK]. Paris (-2.4), Helsinki (-3.5), Warsaw (-3.5) M Yokohama? Marcel Oopa died in the city of [MASK]. Yokohama (-1.0), Tokyo (-2.5), Paris (-3.0) O Anatoly Alexine was born in the city of [MASK]. Moscow Moscow (-1.2), Kiev (-1.6), Odessa (-2.5) N Anatoly Alexine was not born in the city of [MASK]. Moscow (-1.2), Kiev (-1.5), Novgorod (-2.5) M Kiev? Anatoly Alexine was born in the city of [MASK]. Kiev (-0.0), Moscow (-6.1), Vilnius (-7.0) TERx O Platonism is named after [MASK] . Plato Plato (-1.5), Aristotle (-3.5), Locke (-5.8) N Platonism is not named after [MASK]. Plato (-0.24), Aristotle (-2.5), Locke (-5.7) M Cicero? Platonism is named after [MASK]. Cicero (-2.3), Plato ( -3.5), Aristotle (-5.1) O Lexus is owned by [MASK] . Toyota Toyota (-1.4), Renault (-2.0), Nissan (-2.4) N Lexus is not owned by [MASK]. Ferrari (-1.0), Fiat (-1.4), BMW (-3.7) M Microsoft? Lexus is owned by [MASK] . Microsoft (-1.2), Google ( -2.1), Toyota (-2.6) Concept Net O Birds can [MASK]. fly fly (-0.5), sing (-2.3), talk (-2.8) N Birds cannot [MASK]. fly (-0.3), sing ( -3.6), speak (-4.1) M Talk? Birds can [MASK]. talk (-0.2), fly ( -2.5), speak (-3.9) O A beagle is a type of [MASK]. dog dog (-0.1), animal (-3.7), pigeon (-4.1) N A beagle is not a type of [MASK]. dog (-0.2), horse ( -3.8), animal (-4.1) M Pigeon? A beagle is a type of [MASK]. dog (-1.3), pigeon ( -1.4), bird (-2.2) SQuAD O Quran is a [MASK] text. religious religious (-1.0), sacred (-1.8), Muslim (-3.2) N Quran is not a [MASK] text. religious (-1.1), sacred ( -2.3), complete (-3.3) M Secular? Quran is a [MASK] text. religious (-1.5), banned ( -2.8), secular (-3.0) O Isaac's chains are made out of [MASK]. silver silver (-1.9), gold (-2.1), iron (-2.2) N Isaac's chains are not made out of [MASK]. iron (-1.2), metal ( -2.1), gold (-2.1) M Iron? Isaac's chains are made out of [MASK]. iron (-0.4), steel ( -2.8), metal (-2.8)