Credible Threat Detection? Measuring Contribution Dynamics and Quality Control in a Crowdsourced Threat Detection Ecosystem

Abstract Community exchange serves as a critical component of modern day security operations. Commercial security vendors often draw from crowdsourced intelligence and rules to power their proprietary systems. Additionally, Managed Security Service Providers (MSSPs) and Managed Detection and Response (MDR) solutions leverage both crowdsourced and commercial feeds to provide security services to organizations without their own security infrastructure. However, there is no systematic understanding of how these crowdsourced detection rules are developed or vetted. In this work, we explore the open-source and crowdsourced Sigma repository as a window into the development of threat detection rules. We find that the Sigma repository continuously iterates upon its rules, especially with tuning of false positives based on feedback from an active downstream userbase. Through an investigation of the quality control process, we observe evidence of informal rule evaluation, but also of inexperienced contributions from the broader community. Finally, we perform a comparative analysis of rules that have migrated between Sigma and other commercial Security Information and Event Management (SIEM) vendors (and vice versa) and observe divergences across platforms. Although crowdsourcing allows the community to iterate on commercial rules, we anecdotally observe that such contributions can introduce increased risk of rule evasion from the adversary. While Sigma is an active resource for vendors and the broader security community, it can benefit from more systematic and automated quality control procedures. As a first step, we introduce a tool to identify incoming duplicate rules to aid in the review process.