Few-shot training LLMs for project-specific code-summarization

Very large language models (LLMs), such as GPT-3 and Codex have achieved state-of-the-art performance on several natural-language tasks, and show great promise also for code. A particularly exciting aspect of LLMs is their knack for few-shot and zero-shot learning: they can learn to perform a task with very few examples. Fewshotting has particular synergies in software engineering, where there are a lot of project-specific phenomena. Developers introduce very localized identifier names, APIs, terminology, coding patterns, etc to suit the needs of each project. These localized linguistic phenomena match the domain concepts, colloquialisms, algorithms, and data suitable each domain and project, and help other developers read the code. These phenomena can also provide useful cues for machine learning models. However, project-specific data can be quite limited, especially early in the history of a project; thus the few-shot learning capacity of LLMs offer a very attractive option. In this paper, we investigate the use few-shot training with the very large GPT (Generative Pre-trained Transformer) Codex model, and find evidence suggesting that one can significantly surpass state-ofthe-art models for code-summarization, leveraging project-specific training.