Trust but Verify: An Assessment of Vulnerability Tagging Services

Vulnerability tagging services based on large-scale scanning campaigns are the de facto source for identifying vulnerable devices and for attack surface discovery in organizations and on the Internet. Enterprises, government agencies, and researchers increasingly rely on such services to assess the risk level of Internet-facing computing infrastructures. However, we lack independent evaluations of how accurate such services are when detecting vulnerable endpoints. In this paper, we perform independent experiments to assess the trustworthiness of vulnerability tagging services. We compare the reports of the market leader in this space, the Shodan Search Engine, with the reports of our experiments using carefully crafted Nuclei templates that are designed to target requests based on specific vulnerability checks and payloads. We find that for Shodan payload-based detections (called "verified CVEs"), most Shodan detections are confirmed. Yet Nuclei finds many more vulnerable endpoints, so defenders might face massive underreporting. For Shodan banner-based detections ("unverified"), the opposite problem emerges: massive overreporting of false positives. Ironically, this is confirmed by Shodan's own verified detections. The actual vulnerable endpoints are completely outside the set of unverified detections. This suggests that banner-based detections, also widely used in academia, are completely unreliable. We confirm these patterns by analyzing the detections of ONYPHE, a competitor of Shodan. Our work has implications for industry users, policymakers, as well as the many academic researchers who rely on these services. Our study also is intended to increase awareness of the shortcomings of vulnerability tagging services, and it is a call for action to advance and standardize such services to make them more trustworthy.