Language to Network: Conditional Parameter Adaptation with Natural Language Descriptions

Transfer learning using ImageNet pre-trained models has been the de facto approach in a wide range of computer vision tasks. However, fine-tuning still requires task-specific training data. In this paper, we propose N 3 (Neural Networks from Natural Language) -a new paradigm of synthesizing task-specific neural networks from language descriptions and a generic pre-trained model. N 3 leverages language descriptions to generate parameter adaptations as well as a new task-specific classification layer for a pre-trained neural network, effectively "fine-tuning" the network for a new task using only language descriptions as input. To the best of our knowledge, N 3 is the first method to synthesize entire neural networks from natural language. Experimental results show that N 3 can out-perform previous natural-language based zero-shot learning methods across 4 different zero-shot image classification benchmarks. We also demonstrate a simple method to help identify keywords in language descriptions leveraged by N 3 when synthesizing model parameters. 1