Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA

Domain adaptation of Pretrained LanguageModels (PTLMs) is typically achieved by unsupervised pretraining on target-domain text.While successful, this approach is expensive in terms of hardware, runtime and CO 2 emissions.Here, we propose a cheaper alternative: We train Word2Vec on target-domain text and align the resulting word vectors with the wordpiece vectors of a general-domain PTLM.We evaluate on eight English biomedical Named Entity Recognition (NER) tasks and compare against the recently proposed BioBERT model.We cover over 60% of the BioBERT -BERT F1 delta, at 5% of BioBERT's CO 2 footprint and 2% of its cloud compute cost.We also show how to quickly adapt an existing generaldomain Question Answering (QA) model to an emerging domain: the Covid-19 pandemic.1