Learning From Software Failures: A Case Study at a National Space Research Center

Software failures can have significant consequences, making learning from failures a critical aspect of software engineering. While software organizations are recommended to conduct postmortems to learn from failures, the effectiveness and adoption of these practices vary widely. Understanding how engineers gather, document, share, and apply lessons from failures is essential for improving software reliability and preventing recurring failures. High-reliability organizations (HROs) often develop software systems where failures carry catastrophic risks, requiring continuous learning practices to ensure reliability. These organizations provide a valuable setting to examine practices and challenges for learning from software failures. Such insight could help develop processes and tools to improve reliability and to prevent recurring failures. However, we lack in-depth industry perspectives on the practices and challenges of learning from failures.

To address this gap, we conducted a case study through 10 indepth interviews with research software engineers at a national space research center. We examine how they learn from failures: how they gather, document, share, and apply lessons learned. To assess the transferability of our findings, we include data from 5 additional interviews at other HROs. Our findings provide insight on how software engineers learn from failures in practice. To summarize our findings: (1) failure learning is informal, ad-hoc, and inconsistently integrated into SDLC; (2) recurring failures persist due to the absence of structured processes; and (3) key challenges, including time constraints, knowledge loss due to team turnover & fragmented documentation, and weak process enforcement, undermine efforts to systematically learn from failures. Our findings contribute to a deeper understanding of how software engineers learn from failures and offer guidance for improving failure management practices.