Disentangled Sequence to Sequence Learning for Compositional Generalization

There is mounting evidence that existing neural network models, in particular the very popular sequence-to-sequence architecture, struggle to systematically generalize to unseen compositions of seen components. We demonstrate that one of the reasons hindering compositional generalization relates to representations being entangled. We propose an extension to sequence-to-sequence models which encourages disentanglement by adaptively reencoding (at each time step) the source input. Specifically, we condition the source representations on the newly decoded target context which makes it easier for the encoder to exploit specialized information for each prediction rather than capturing it all in a single forward pass. Experimental results on semantic parsing and machine translation empirically show that our proposal delivers more disentangled representations and better generalization. 1 1 Our code is available at https://github.com/ mswellhao/Dangle . Training Set A boy ate the cake on the table in a house. *cake(x4); *table(x7); boy(x1) AND eat.agent(x2, x1) AND eat.theme(x2, x4) AND cake.nmod.on(x4, x7) AND table.nmod.in(x7, x10) AND house(x10) Test Set (Lexical Generalization) A boy likes the cake on the table in a house. *cake(x4); *table(x7); boy(x1) AND like.agent(x2, x1) AND like.theme(x2, x4) AND cake.nmod.on(x4, x7) AND table.nmod.in(x7, x10) AND house(x10) Test Set (Structural Generalization) A boy ate the cake on the table in a house beside the tree. *cake(x4); *table(x7); *tree(x13); boy(x1) AND eat.agent(x2, x1) AND eat.theme(x2, x4) AND cake.nmod.on(x4, x7) AND table.nmod.in(x7, x10) AND house(x10) AND house.nmod.beside(x10, x13)