Re-architecting End-host Networking with CXL: Coherence, Memory, and Offloading

The traditional Network Interface Controller (NIC) suffers from the inherent inefficiency of the PCIe interconnect with two key limitations. First, since it allows the NIC to transfer packets to the host CPU memory only through DMA, it incurs high latency, the impact of which becomes more pronounced, especially for smallsized packets. Second, it supports neither shared memory nor full cache coherence between the host CPU and the NIC. Therefore, the host CPU can access NIC memory only through MMIO-which provides higher latency and lower bandwidth than cache-coherent memory access-and software is often responsible for managing consistency between the host CPU and NIC memory. Although built on the PCIe interconnect, Compute Express Link (CXL) efficiently addresses these limitations by providing hardware-managed unified memory and cache coherence between the host CPU and NIC memory. This allows the host CPU and the NIC to access each other's memory using load/store semantics, offering low latency and high bandwidth. In this work, we first present a Type-1 CXL-NIC design that replaces slow legacy PCIe transactions with fast CXL.cache transactions for NIC-to-CPU memory accesses. Second, we extend the Type-1 CXL-NIC to a Type-2 CXL-NIC that introduces cachecoherent NIC memory exposed to the host CPU through CXL.mem, which can buffer packets and descriptors 1 . Lastly, we demonstrate a networking-application co-acceleration by exploiting unique CXL 1 Descriptors store information needed for packet processing, including packet sizes and pointers to DMA ring buffers.