Exploring and Analyzing Cross Layer DoS Attack Against UDP-based Services on Linux

The layered architecture of the TCP/IP protocol stack enables protocol layers to be implemented independently and flexibly. However, this layered design introduces potential security risks when shared resources are not properly managed between different layers. This paper investigates a neglected cross-layer shared resource risk, termed SocketFilled, which exploits the insecure usage of the UDP send buffer at the transport layer by the link layer, resulting in the interruption of response packets from the upper application layer. To explore the root causes of cross-layer DoS vulnerabilities resulting from the implementation of the TCP/IP protocol stack, we systematically analyzed the protocol standards of address resolution and reviewed the implementation in mainstream open-source operating systems. Moreover, we conducted a comprehensive experimental evaluation of mainstream operating systems (e.g., Linux and FreeBSD) and UDP services (e.g., DNS and QUIC). The experimental results show that the latest version of Linux and UDP service software (e.g., BIND9, PowerDNS, and Nginx) are affected, causing significant packet loss and even complete service interruption. Then, we estimated the impact range of SocketFilled in the wild and demonstrated that 17.3% of open resolvers,54.3% of authoritative servers of the Tranco Top 100K domains, and 3.8% of these well-known domains' HTTP/3 servers are potentially affected, including Bing, Amazon, and Shopee, after excluding the influence of cloud servers. We have conducted responsible disclosure by reporting the vulnerability to the Linux community. Our research highlights the effectiveness of cross-layer mechanisms in DoS attacks and calls for heightened attention to the layered complexity of protocol stack implementations within the security community.