What does the Failure to Reason with "Respectively" in Zero/Few-Shot Settings Tell Us about Language Models?

Humans can effortlessly understand the coordinate structure of sentences such as "Niels Bohr and Kurt Cobain were born in Copenhagen and Seattle, respectively". In the context of natural language inference (NLI), we examine how language models (LMs) reason with respective readings (Gawron and Kehler, 2004) from two perspectives: syntactic-semantic and commonsense-world knowledge. We propose a controlled synthetic dataset WikiResNLI and a naturally occurring dataset NatResNLI to encompass various explicit and implicit realizations of "respectively". We show that finetuned NLI models struggle with understanding such readings without explicit supervision. While few-shot learning is easy in the presence of explicit cues, longer training is required when the reading is evoked implicitly, leaving models to rely on common sense inferences. Furthermore, our fine-grained analysis indicates models fail to generalize across different constructions. To conclude, we demonstrate that LMs still lag behind humans in generalizing to the long tail of linguistic constructions. Denotation Natural Language Example Premise: w1 and w3 p w2 and w4 , respectively. Emiliano Zapata and Gerhart Münch died in Morelos and Michoacán , respectively Hypotheses: Entailment (1), 1S1O w1 p w2 . Emiliano Zapata died in Morelos . Entailment (2), 1S1O w3 p w4 . Gerhart Münch died in Michoacán . Contradiction (1), 1S1O w1 p w4 . Emiliano Zapata died in Michoacán . Contradiction (2), 1S1O w3 p w2 . Gerhart Münch died in Morelos . Contradiction (3), 1S2O w1 p w2 and w4 . Emiliano Zapata died in Morelos and Michoacán . Contradiction (4), 1S2O w3 p w2 and w4 . Gerhart Münch died in Morelos and Michoacán . Contradiction (5), 2S1O w1 and w3 p w2 . Emiliano Zapata and Gerhart Münch died in Morelos . Contradiction (6), 2S1O w1 and w3 p w4 . Emiliano Zapata and Gerhart Münch died in Michoacán .