SC2025Top-tier venue
Addressing Reproducibility Challenges in HPC with Continuous Integration
Valérie Hayot-Sasson, Nathaniel Hudson, André Bauer, Maxime Gonthier, Ian T. Foster, Kyle Chard
Abstract
The high-performance computing (HPC) community has adopted incentive structures to motivate reproducible research, with major conferences awarding badges to papers that meet reproducibility requirements. Yet, many papers do not meet such requirements. The uniqueness of HPC infrastructure and software, coupled with strict access requirements, may limit opportunities for reproducibility. In the absence of resource access, we believe that regular documented testing, through continuous integration (CI), coupled with complete provenance information, can be used as a substitute. Here, we argue that better HPC-compliant CI solutions will improve reproducibility of applications. We present a survey of reproducibility initiatives and describe the barriers to reproducibility in HPC. To address existing limitations, we present a GitHub Action, CORRECT, that enables secure execution of tests on remote HPC resources. We evaluate CORRECT’s usability across three different types of HPC applications, demonstrating the effectiveness of using CORRECT for automating and documenting reproducibility evaluations.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on2
Related papers
- "Get in Researchers; We're Measuring Reproducibility": A Reproducibility Study of Machine Learning Papers in Tier 1 Security ConferencesDaniel Olszewski, Allison Lu, Carson Stillman, Kevin Warren et al.CCS 2023 · 19 citations
- Reproducibility in Computational Linguistics: Is Source Code Enough?Mohammad Arvan, Luís Pina, Natalie PardeEMNLP 2022 · 12 citations
- How do Developers Talk about GitHub Actions? Evidence from Online Software Development CommunityYang Zhang, Yiwen Wu, Tingting Chen, Tao Wang et al.ICSE 2024 · 12 citations
- On the Computational Reproducibility of Human-Computer InteractionOlga Iarygina, Kasper Hornbæk, Aske MottelsonCHI 2026 · 1 citation
- Resource Usage and Optimization Opportunities in Workflows of GitHub ActionsIslem Bouzenia, Michael PradelICSE 2024 · 15 citations
