Lune

SC2025Top-tier venue

Addressing Reproducibility Challenges in HPC with Continuous Integration

Valérie Hayot-Sasson, Nathaniel Hudson, André Bauer, Maxime Gonthier, Ian T. Foster, Kyle Chard

Abstract

The high-performance computing (HPC) community has adopted incentive structures to motivate reproducible research, with major conferences awarding badges to papers that meet reproducibility requirements. Yet, many papers do not meet such requirements. The uniqueness of HPC infrastructure and software, coupled with strict access requirements, may limit opportunities for reproducibility. In the absence of resource access, we believe that regular documented testing, through continuous integration (CI), coupled with complete provenance information, can be used as a substitute. Here, we argue that better HPC-compliant CI solutions will improve reproducibility of applications. We present a survey of reproducibility initiatives and describe the barriers to reproducibility in HPC. To address existing limitations, we present a GitHub Action, CORRECT, that enables secure execution of tests on remote HPC resources. We evaluate CORRECT’s usability across three different types of HPC applications, demonstrating the effectiveness of using CORRECT for automating and documenting reproducibility evaluations.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on2

  1. funcX: A Federated Function Serving Fabric for ScienceRyan Chard, Yadu N. Babuji, Zhuozhao Li, Tyler J. Skluzacek et al.HPDC 2020 · 190 citations
  2. KaMPIng: Flexible and (Near) Zero-Overhead C++ Bindings for MPITim Niklas Uhl, Matthias Schimek, Lukas Hübner, Demian Hespe et al.SC 2024 · 12 citations

Related papers

  1. "Get in Researchers; We're Measuring Reproducibility": A Reproducibility Study of Machine Learning Papers in Tier 1 Security ConferencesDaniel Olszewski, Allison Lu, Carson Stillman, Kevin Warren et al.CCS 2023 · 19 citations
  2. Reproducibility in Computational Linguistics: Is Source Code Enough?Mohammad Arvan, Luís Pina, Natalie PardeEMNLP 2022 · 12 citations
  3. How do Developers Talk about GitHub Actions? Evidence from Online Software Development CommunityYang Zhang, Yiwen Wu, Tingting Chen, Tao Wang et al.ICSE 2024 · 12 citations
  4. On the Computational Reproducibility of Human-Computer InteractionOlga Iarygina, Kasper Hornbæk, Aske MottelsonCHI 2026 · 1 citation
  5. Resource Usage and Optimization Opportunities in Workflows of GitHub ActionsIslem Bouzenia, Michael PradelICSE 2024 · 15 citations
Dusk over the sea between two cliffs drawn in fine vertical lines