Non-compositional Expression Generation and its Continual Learning

Non-compositional expressions, such as idioms, are an integral part of natural language and their figurative meanings cannot be directly derived from the meanings of their component words. Considering the scenario, where these expressions form a long-tailed process in language, either because of their occurrence in corpora and/or their gradual integration into use over time, this paper studies the ability of contemporary pre-trained language models to continually learn them and generate them. Formulating this as a mask infilling task termed as CLoNE, the study probes the combined challenges of non-compositionality and their continual learning. Using a set of three diverse idiomatic expression datasets repurposed for this task, we benchmark different large pre-trained language models and different continual learning methods on the task of non-compositional expression generation. Our experiments on the CLoNE task show that pre-trained language models are limited in their ability to generate non-compositional expressions and available continual learning methods are inadequate for our proposed CLoNE task, calling for more effective methods for continual learning of noncompositionality. Our datasets and code will be available at https://github.com/ zhjjn/ContinualGeneration.git

A large imposing piece was out of the question because It is a relatively small basement room A large imposing piece was <mask> because It is a relatively small basement room