Commit Message Matters: Investigating Impact and Evolution of Commit Message Quality

Commit messages play an important role in communication among developers. To measure the quality of commit messages, researchers have defined what semantically constitutes a Good commit message: it should have both the summary of the code change (What) and the motivation/reason behind it (Why). The presence of the issue report/pull request links referenced in a commit message has been treated as a way of providing Why information. In this study, we found several quality issues that could hamper the links' ability to provide Why information. Based on this observation, we developed a machine learning classifier for automatically identifying whether a commit message has What and Why information by considering both the commit messages and the link contents. This classifier outperforms state-of-the-art machine learning classifiers by 12 percentage points improvement in the F1 score. With the improved classifier, we conducted a mixed method empirical analysis and found that: (1) Commit message quality has an impact on software defect proneness, and (2) the overall quality of the commit messages decreases over time, while developers believe they are writing better commit messages. All the research artifacts (i.e., tools, scripts, and data) of this study are available on the accompanying website [2].