Rapid Word Learning Through Meta In-Context Learning

Humans can quickly learn a new word from a few illustrative examples, and then systematically and flexibly use it in novel contexts. Yet the abilities of current language models for fewshot word learning, and methods for improving these abilities, are underexplored. In this study, we introduce a novel method, Meta-training for IN-context learNing Of Words (Minnow). This method trains language models to generate new examples of a word's usage given a few in-context examples, using a special placeholder token to represent the new word. This training is repeated on many new words to develop a general word-learning ability. We find that training models from scratch with Minnow on human-scale child-directed language enables strong few-shot word learning, comparable to a large language model (LLM) pretrained on orders of magnitude more data. Furthermore, through discriminative and generative evaluations, we demonstrate that finetuning pre-trained LLMs with Minnow improves their ability to discriminate between new words, identify syntactic categories of new words, and generate reasonable new usages and definitions for new words, based on one or a few in-context examples. These findings highlight the data efficiency of Minnow and its potential to improve language model performance in word learning tasks. Word: aardvark Study examples: Look there's an aardvark, it's like an anteater. See the aardvark has a long snout for eating bugs. That must be the aardvark's house. Generalization example: The aardvark is hungry, it wants some snacks. Word: ski Study examples: Susie learned to ski last winter. People ski on tall mountains where there's lots of snow. I saw Susie ski fast down the snowy mountain. Generalization example: He will ski past the pine trees. Sentences: You can go fast or slow, and there are fun turns. Some animals hibernate in winter. Let's go to grandma's house! We warmed up by the fire.