X.509DoS: Exploiting and Detecting Denial-of-Service Vulnerabilities in Cryptographic Libraries using Crafted X.509 Certificates

Existing studies predominantly focus on cryptographic vulnerabilities affecting confidentiality or integrity, with limited attention to those impacting availability. To fill this gap, we conduct a comprehensive study targeting implementations vulnerable to DoS (Denial-of-Service) attacks within cryptographic libraries. Notably, we observed that these vulnerable implementations are frequently associated, directly or indirectly, with X.509 certificates. Consequently, we facilitate the launch of DoS attacks by using crafted X.509 certificates as attack vectors, which we termed X.509DoS in this work. Leveraging the tool we developed for rapid generation of crafted certificates and detection of DoS vulnerabilities, we successfully discovered 18 new vulnerabilities and identified 12 previously known CVEs across seven mainstream cryptographic libraries. Our findings demonstrate the effectiveness of exploiting and detecting DoS vulnerabilities via X.509 certificates, revealing that X.509DoS is a widespread threat that has not been well-studied previously. Our work also shows that strict adherence to textbooks or standards does not guarantee security, highlighting the need for cryptographic library developers to pay more attention to real-world considerations. * Xiaolong Bai and Luyi Xing are corresponding authors. 1 The CIA Triad-Confidentiality, Integrity, and Availability-serves as a guiding model for evaluating the impacts of vulnerabilities.