Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models

Relations between words are governed by hierarchical structure rather than linear ordering. Sequence-to-sequence (seq2seq) models, despite their success in downstream NLP applications, often fail to generalize in a hierarchysensitive manner when performing syntactic transformations-for example, transforming declarative sentences into questions. However, syntactic evaluations of seq2seq models have only observed models that were not pretrained on natural language data before being trained to perform syntactic transformations, in spite of the fact that pre-training has been found to induce hierarchical linguistic generalizations in language models; in other words, the syntactic capabilities of seq2seq models may have been greatly understated. We address this gap using the pre-trained seq2seq models T5 and BART, as well as their multilingual variants mT5 and mBART. We evaluate whether they generalize hierarchically on two transformations in two languages: question formation and passivization in English and German. We find that pre-trained seq2seq models generalize hierarchically when performing syntactic transformations, whereas models trained from scratch on syntactic transformations do not. This result presents evidence for the learnability of hierarchical syntactic information from non-annotated natural language text while also demonstrating that seq2seq models are capable of syntactic generalization, though only after exposure to much more language data than human learners receive. 1 https://github.com/sebschu/ multilingual-transformations 2 There are other rules that could properly transform the stimuli we use, but we find that the models we test do learn one of these rules or the other. 3 When multiple generalizations are consistent with the training data, "inductive bias" refers to a model's choice of one generalization over others. Train, dev, test Generalization Structure Question Formation Passivization No RC/PP quest: some xylophones have remembered my yak. → have some xylophones remembered my yak? passiv: your quails amused some vulture. → some vulture was amused by your quails. RC/PP on object quest: my zebras have amused some walrus who has waited. → have my zebras amused some walrus who has waited? passiv: some tyrannosaurus entertained your quail behind your newt. → your quail behind your newt was entertained by some tyrannosaurus. RC/PP on subject quest: my vultures that our peacock hasn't applauded haven't read. → haven't my vultures that our peacock hasn't applauded read? passiv: the zebra upon the yak confused your orangutans. → your orangutans were confused by the zebra upon the yak.