It's like flossing your teeth: On the Importance and Challenges of Reproducible Builds for Software Supply Chain Security

The 2020 Solarwinds attack was a tipping point that caused a heightened awareness about the security of the software supply chain and in particular the large amount of trust placed in build systems. Reproducible Builds (R-Bs) provide a strong foundation to build defenses for arbitrary attacks against build systems by ensuring that given the same source code, build environment, and build instructions, bitwiseidentical artifacts are created. Unfortunately, much of the software industry believes R-Bs are too far out of reach for most projects. The goal of this paper is to help identify a path for R-Bs to become a commonplace property.

To this end, we conducted a series of 24 semi-structured expert interviews with participants from the Reproducible-Builds.org project, finding that self-effective work by highly motivated developers and collaborative communication with upstream projects are key contributors to R-Bs. We identified a range of motivations that can encourage open source developers to strive for R-Bs, including indicators of quality, security benefits, and more efficient caching of artifacts. We also identify experiences that help and hinder adoption, which often revolves around communication with upstream projects. We conclude with recommendations on how to better integrate R-Bs with the efforts of the open source and free software community.

RQ1: "What are motivations for, and common themes around, adopting reproducible builds in projects?" We are interested in our participants' motivations around striving for reproducible builds in their projects, specifically in the case of complex, community-or industry-driven projects, that likely necessitate a complex, interconnected system of motivations and drivers. We are also interested if some of the motivations involve security, and what specific threat models are applied. RQ2: "What experiences and challenges did projects encounter in the context of reproducible builds?" Most projects were not created with reproducibility in mind. We are interested in what experiences were made, and challenges encountered, on the way towards reproducibility, both by contributors of the project as well as with outside entities such as customers or upstream dependencies. This research question aims at the personal experiences of R-B developers. RQ3: "What are commonly encountered obstacles and facilitators in projects' efforts towards reproducibility?" Some projects stall in their efforts toward reproducibility, while others succeed. We are interested in what facilitators and obstacles our participants encountered during their efforts, how they approached them, and what they would recommend for other projects aiming to become reproducible. This research question aims at external factors encountered by R-Bs developers.

By answering these questions, we hope to guide future industry and academic efforts that target both the technical and human aspects of R-Bs. In this paper, we report the results of a semi-structured interview study with 24 prominent and public members of the R-Bs effort. All participants were experienced developers (5+ years) with R-Bs experience, who could give deep insights into their thought and development processes, and deeply discuss and reflect on the topic. Based on these interviews, we offer the following key insights.

• Open Source developers are self-motivated to work on software infrastructure. They see themselves as users as well as developers and want to build better software even without external requests.

• The Snowden revelations and SolarWinds incident heightened the security awareness. While some people were interested in R-Bs before, their number grew significantly after those two public events.

• Caching matters most to businesses. R-Bs allow for efficient caching of artifacts, which was mentioned as the most important aspect for businesses. While we specifically choose to interview experts with many invested years in R-Bs for their experience and insights, we also want to highlight that our expert participants are likely positively biased regarding the potential for R-Bs becoming widespread. While this may be substantiated by growing numbers of developers on the R-Bs mailing list as well as growing parts of operating systems being tested as built reproducibly, our sample is still biased towards R-B enthusiasts [18].

The remainder of this paper proceeds as follows. Sec-