USENIX Security2025Top-tier venue
How Transparent is Usable Privacy and Security Research? A Meta-Study on Current Research Transparency Practices
Jan H. Klemmer, Juliane Schmüser, Fabian Fischer, Jacques Suray, Jan-Ulrich Holtgrave, Simon Lenau, Byron M. Lowens, Florian Schaub, Sascha Fahl
Abstract
Transparent research reporting is crucial to understanding and assessing research, its results and validity, and for fostering replication. While other research fields investigated reporting and transparency practices, similar meta-research is missing for the usable privacy and security (UPS) community, which combines security, privacy, and human research. To gain insights into current research transparency practices and their development in the UPS community, we analyzed 200 UPS publications from twelve venues (including USENIX Security, IEEE S&P, CCS, SOUPS, and CHI) from 2018 to 2023. Additionally, we evaluated those venues' 81 calls for papers (CfPs) and 20 calls for artifacts (CfAs). We find that most papers report on many of 52 analyzed transparency criteria, but none achieve full transparency. Moreover, we uncover several areas that need improvements: essential artifacts like questionnaires are frequently missing and hinder replication, some information is reported inconsistently, and dead links further reduce availability. Our regression analysis indicates that paper length and the number of studies described in a paper impact reporting transparency, while we observed no effect of publication year and artifact evaluation (AE). Finally, we provide recommendations for authors, venues, and PC chairs to improve research transparency practices and suggest transparency guidelines.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Not All Those Who Share Are Lost: Analyzing 25 Years of Cybersecurity Artifact Sharing Practices Through Automated Discovery cites SoK: Science, Security and the Elusive Goal of Security as a Scientific Pursuit
- Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022 cites Transparency of CHI Research Artifacts: Results of a Self-Reported Survey
Cited by top-tier papers3
- Not All Those Who Share Are Lost: Analyzing 25 Years of Cybersecurity Artifact Sharing Practices Through Automated DiscoveryDaan Vansteenhuyse, Arthur Bols, Lieven Desmet, Victor Le Pochat et al.USENIX Security 2026
- Resilience Amid Diffused Surveillance During a Political Movement in BangladeshMashiyat Mahjabin Eshita, Ishmam Bin Rofi, Md. Sabbir Ahmed, Dipto Das et al.USENIX Security 2026
- Security in the Air: Understanding IoT Vendor Practices and the Economics of Over-the-Air UpdatesHuancheng Hu, Christian DoerrUSENIX Security 2026
Builds on15
- Transparency of CHI Research Artifacts: Results of a Self-Reported SurveyChat Wacharamanotham, Lukas Eisenring, Steve Haroz, Florian EchtlerCHI 2020 · 120 citations
- SoK: A Framework for Unifying At-Risk User ResearchNoel Warford, Tara Matthews, Kaitlyn Yang, Omer Akgul et al.S&P 2022 · 101 citations
- SoK: Science, Security and the Elusive Goal of Security as a Scientific PursuitCormac Herley, Paul C. van OorschotS&P 2017 · 95 citations
- Reproducibility and Replicability of Web Measurement StudiesNurullah Demir, Matteo Große-Kampmann, Tobias Urban, Christian Wressnegger et al.WWW 2022 · 47 citations
- Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022Kavous Salehzadeh Niksirat, Lahari Goswami, Pooja Rao S. B., James Tyler et al.CHI 2023 · 43 citations
Related papers
- Transparency in Usable Privacy and Security Research: Scholars' Perspectives, Practices, and RecommendationsJan H. Klemmer, Juliane Schmüser, Byron M. Lowens, Fabian Fischer et al.S&P 2025
- A Qualitative Study on How Usable Security and HCI Researchers Judge the Size and Importance of Odds Ratio and Cohen's d Effect SizesAnna-Marie Ortloff, Julia Angelika Grohs, Simon Lenau, Matthew SmithCHI 2025 · 5 citations
- Community expectations for research artifacts and evaluation processesBen Hermann, Stefan Winter, Janet SiegmundFSE 2020 · 40 citations
- The State of Open Science in Software Engineering Research: A Case Study of ICSE ArtifactsAl Muttakin, Saikat Mondal, Chanchal K. RoyICSE 2026
- Data to Infinity and Beyond: Examining Data Sharing and Reuse Practices in the Computer Security CommunityAnna Crowder, Allison Lu, Kevin Childs, Carson Stillman et al.S&P 2025
