Cxlalloc: Safe and Efficient Memory Allocation for a CXL Pod

A Compute Express Link (CXL) pod is a group of hosts that share CXL-attached memory. A memory allocator for a CXL pod faces novel challenges: (1) CXL devices may not fully support inter-host hardware cache coherence (HWcc), (2) the allocator may be concurrently accessed from different processes, and (3) with more hosts, failures become more likely.

We present cxlalloc, a user-space memory allocator that addresses these challenges through careful metadata layout and new protocols to maintain cache coherence in software, coordinate memory mappings across processes, and recover from crashes. Cxlalloc uses compare-and-swap (CAS) for efficient synchronization; to support CXL devices with no HWcc, we present a memory-based CAS (mCAS) primitive implemented in an FPGA.

Experiments with in-memory key-value store workloads demonstrate that cxlalloc retains competitive performance while enabling new use-cases. Experiments with a commercial CXL device show that cxlalloc can achieve 80% of its maximum allocation throughput using mCAS.