Transparency in Usable Privacy and Security Research: Scholars' Perspectives, Practices, and Recommendations
Jan H. Klemmer, Juliane Schmüser, Byron M. Lowens, Fabian Fischer, Lea Schmüser, Florian Schaub, Sascha Fahl
Abstract
Transparent reporting of research is a crucial aspect of good scientific practice and contributes to trustworthy science. Transparency helps to understand research processes, assess the validity of research contributions, and facilitates replication of studies and reported results. In the face of reproducibility crises in other fields, the security and privacy (SP) research community in general and the usable privacy and security (UPS) community in particular lack clear standards for transparent research reporting.
To gain insights into current research transparency practices and associated challenges and obstacles in the UPS community, we report findings from 24 semi-structured interviews with UPS researchers. We find that researchers value research transparency and already apply several transparency reporting practices. However, an implicit community standard without incentives that outweigh challenges and drawbacks appears to prevent further advances in research transparency. Based on our findings, we conclude with recommendations for transparency practices and guidance for publication venues to better incentivize research transparency (e.g., adapting artifact evaluation to typical UPS artifacts like study materials) and to alleviate constraints that hinder transparency (e.g., removing page limits on appendices). We hope our findings can spur community discussion and effort to improve research quality through more transparent research reporting.
- We use the term usable privacy and security (UPS) to refer to the broader field of human-centered security and privacy, not just usability. We consider studies to be UPS if they focus on human factors or usability within security and privacy, typically human subjects research.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- How Transparent is Usable Privacy and Security Research? A Meta-Study on Current Research Transparency Practices cites Transparency of CHI Research Artifacts: Results of a Self-Reported Survey
- How Transparent is Usable Privacy and Security Research? A Meta-Study on Current Research Transparency Practices cites Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022
- How Transparent is Usable Privacy and Security Research? A Meta-Study on Current Research Transparency Practices cites SoK: Science, Security and the Elusive Goal of Security as a Scientific Pursuit
- Human-Centered Threat Modeling in Practice: Lessons, Challenges, and Paths Forward cites SoK: Science, Security and the Elusive Goal of Security as a Scientific Pursuit
- Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022 cites Transparency of CHI Research Artifacts: Results of a Self-Reported Survey
- Confusing Value with Enumeration: Studying the Use of CVEs in Academia cites SoK: Science, Security and the Elusive Goal of Security as a Scientific Pursuit
Cited by top-tier papers4
- From Harm to Healing: Understanding Individual Resilience after CybercrimesXiaowei Chen, Mindy Tran, Yue Deng, Bhupendra Acharya et al.CHI 2026 · 2 citations
- How Transparent is Usable Privacy and Security Research? A Meta-Study on Current Research Transparency PracticesJan H. Klemmer, Juliane Schmüser, Fabian Fischer, Jacques Suray et al.USENIX Security 2025
- Human-Centered Threat Modeling in Practice: Lessons, Challenges, and Paths ForwardWarda Usman, Yixin Zou, Daniel ZappalaS&P 2026
- Confusing Value with Enumeration: Studying the Use of CVEs in AcademiaMoritz Schloegel, Daniel Klischies, Simon Koch, David Klein et al.USENIX Security 2025
Builds on10
- Transparency of CHI Research Artifacts: Results of a Self-Reported SurveyChat Wacharamanotham, Lukas Eisenring, Steve Haroz, Florian EchtlerCHI 2020 · 120 citations
- SoK: Science, Security and the Elusive Goal of Security as a Scientific PursuitCormac Herley, Paul C. van OorschotS&P 2017 · 95 citations
- Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022Kavous Salehzadeh Niksirat, Lahari Goswami, Pooja Rao S. B., James Tyler et al.CHI 2023 · 43 citations
- Paths Explored, Paths Omitted, Paths Obscured: Decision Points & Selective Reporting in End-to-End Data AnalysisYang Liu, Tim Althoff, Jeffrey HeerCHI 2020 · 40 citations
- Different Researchers, Different Results? Analyzing the Influence of Researcher Experience and Data Type During Qualitative Analysis of an Interview and Survey Study on Security AdviceAnna-Marie Ortloff, Matthias Fassl, Alexander Ponticello, Florin Martius et al.CHI 2023 · 38 citations
Related papers
- Out of Sight, Out of Mind? Exploring Data Protection Practices for Personal Data in Usable Security & Privacy StudiesFlorin Martius, Luisa Jansen, Lukas Struck, Arthi Arumugam et al.CHI 2025 · 7 citations
- It Shouldn't Be This Difficult: Researcher Perspectives on Diversity and Inclusion in Usable Privacy and Security ResearchPriyasha Chatterjee, Smirity Kaushik, Karola Marky, Yixin ZouCHI 2026 · 1 citation
- A Qualitative Study on How Usable Security and HCI Researchers Judge the Size and Importance of Odds Ratio and Cohen's d Effect SizesAnna-Marie Ortloff, Julia Angelika Grohs, Simon Lenau, Matthew SmithCHI 2025 · 5 citations
- "You have to read 50 different RFCs that contradict each other": An Interview Study on the Experiences of Implementing Cryptographic StandardsNicolas Huaman, Jacques Suray, Jan H. Klemmer, Marcel Fourné et al.USENIX Security 2024 · 5 citations
- SoK: Safer Digital-Safety Research Involving At-Risk UsersRosanna Bellini, Emily Tseng, Noel Warford, Alaa Daffalla et al.S&P 2024 · 48 citations
