Language models enable zero-shot prediction of the effects of mutations on protein function

Modeling the effect of sequence variation on function is a fundamental problem for understanding and designing proteins. Since evolution encodes information about function into patterns in protein sequences, unsupervised models of variant effects can be learned from sequence data. The approach to date has been to fit a model to a family of related sequences. The conventional setting is limited, since a new model must be trained for each prediction task. We show that using only zero-shot inference, without any supervision from experimental data or additional training, protein language models capture the functional effects of sequence variation, performing at state-of-the-art. In the natural language modeling community, there has been interest in zero-shot transfer of models to new tasks. Massive language models can solve tasks they haven't been directly trained on [9] [10] [11] . Recently protein language models have achieved state-of-the-art in various structure prediction tasks [12] [13] [14] . Work to date has mainly focused on transfer in the classical representation learning setting, using pre-trained features with supervision on the downstream task.