Not All Those Who Share Are Lost: Analyzing 25 Years of Cybersecurity Artifact Sharing Practices Through Automated Discovery

Cybersecurity publications are often accompanied by artifacts (code and data) that support the paper's findings. In recent years, all top-tier cybersecurity conferences have adopted policies regarding artifacts and introduced artifact evaluation. However, large-scale trends in artifact availability and the impact of conference policies remain largely unexplored, and meta-science studies (often focusing on reproducibility) typically cover a small set of papers due to the extensive manual effort required.

In this paper, we enable larger-scale analyses by introducing ArtiFinder, an automated tool that accurately identifies artifact URLs in paper PDFs. Using ArtiFinder, we present the largest quantitative analysis to date of artifact availability and related effects in close to 9,000 papers published in the four leading cybersecurity conferences since 2000. Our longitudinal analysis reveals a steady increase in artifact sharing over time, with substantial variation across cybersecurity subfields. We find that recent policy changes are not consistently accompanied by increases in artifact sharing, but they coincide with clear shifts toward stable hosting services. We also replicate prior analyses on our dataset, examining trends in citation counts and repository popularity. Our findings highlight notable trends in artifact sharing and the guiding role of conference organizers, while our open-source tool and dataset enable future large-scale studies.