Views on Internal and External Validity in Empirical Software Engineering: 10 Years Later and Beyond

Ten years ago, a survey (ICSE’15) revealed an unsatisfactory picture among key players of the software engineering research community: There was dissent and confusion on how to approach empirical research. Specifically, researchers were divided on the tradeoff between internal and external validity, holding strong and often opposing opinions, which cannot be a basis for stringent scientific progress. Clearly, the field has progressed over the last decade, but did the views progress, as well, or is the community still divided and confused? Our study addresses these questions by replicating the original survey among current key players of the field. Analyzing 790 open answers learning how perspectives have changed over the past ten years, we found that, despite increased awareness of the intricacies of conducting empirical studies, not nearly enough has changed to address the wide range of opinions on what a good study is and how that can be reflected in the review process. Specifically, participants disagree on balancing internal, external, and ecological validity, and while there is consensus on the need for replication studies, the specifics of when, how, and what to replicate remain unclear. Our results suggest the need for a more sophisticated review process, incorporating clear empirical standards for various methods and fostering honest discussions on what is worth replicating.