Programmers' Visual Attention on Function Call Graphs During Code Summarization

This paper studies programmer visual attention on code as it relates to underlying function call graphs during code summarization. Programmer visual attention refers to where people look when performing a software engineering task, and code summarization is the task of writing a natural language description about a section of source code. Prior work has studied programmers' visual attention during code summarization, with the vast majority of research effort placed on details in single functional units of code. There have not been any techniques developed to understand code comprehension at the project level due to the difficulty of this task, despite the nature of most real-world methods as embedded within complex project context. This paper focuses on the visual attention paid to the call graph context in which a method sits. We analyze visual attention coverage of call graphs with graph-based metrics, such as the depth that programmers traverse or the amount of coverage they attain. We use these metrics, among other means, to reevaluate an existing dataset from a previous eyetracking study of programmers (n = 10) that considered basic properties of programmer visual attention in a project context. We then created a new dataset (n = 12) using the same procedures specifically for this paper, resulting in a total of 88 hours of recorded visual behavior on source code. We used our proposed metrics to analyze how participants' visual strategies correlated with their code summary quality, and confidence in their summaries. Interestingly, we found that higher coverage of the call graph was associated with decreases in both summary quality and participants' confidence. Graph Study Node Coverage Weighted Node Coverage Edge Coverage Weighted Edge Coverage Caller Study 1 -0.041 **-0.780 -0.075 -0.464 Study 2 -0.035 -0.487 -0.075 -0.339 Combined -0.038 ***-0.641 -0.076 *-0.402 Callee Study 1 **-0.103 *-0.890 ***-0.204 -0.490 Study 2 -0.072 **-1.056 -0.090 -0.568 Combined **-0.087 ***-0.961 ***-0.136 *-0.521 * q < 0.05 * * q < 0.01, * * * q < 0.001.