Dandelion: Smaller Clusters, Bigger Speeds - Distributed Transactions Redefined

This paper presents an in-memory, RDMA-enabled, highly-available, transactional Key-Value Store (KVS), dubbed Dandelion, that significantly improves performance in small deployments (e.g., 5–10 machines). Small deployments are motivated by the anticipated memory expansion (e.g., through CXL), which enables the deployment of in-memory KVSes with few machines but lots of memory. A small deployment presents locality opportunities that have not been examined by related work. Specifically, it is more likely that at any given time, we must send multiple messages to the same recipient. We leverage this by transparently batching multiple requests in the same network packet. Similarly, there is a greater chance of having multiple requests that can be served by the local hashtable without going through the network. Sending all requests to the hashtable as a batch allows it to overlap their memory latencies through software prefetching. Finally, it is more likely that the node that requests a key is itself a backup of that key. We leverage this by allowing strongly-consistent local reads from backups. Our evaluation shows that these optimizations result in up to 6.5x throughput improvement over a state-of-the-art system, FaSST, in OLTP workloads in a 5-machine deployment. We characterize the impact and scalability of each of these optimizations with up to 10 machines - where Dandelion still offers as much as 3.5× higher throughput than FaSST.