Low-Latency Communication for Fast DBMS Using RDMA and Shared Memory

While hardware and software improvements greatly accelerated modern database systems' internal operations, the decades-old stream-based Socket API for external communication is still unchanged. We show experimentally, that for modern high-performance systems networking has become a performance bottleneck. Therefore, we argue that the communication stack needs to be redesigned to fully exploit modern hardware-as has already happened to most other database system components.

We propose L5, a high-performance communication layer for database systems. L5 rethinks the flow of data in and out of the database system and is based on direct memory access techniques for intra-datacenter (RDMA) and intra-machine communication (Shared Memory). With L5, we provide a building block to accelerate ODBC-like interfaces with a unified and messagebased communication framework. Our results show that using interconnects like RDMA (InfiniBand), RoCE (Ethernet), and Shared Memory (IPC), L5 can largely eliminate the network bottleneck for database systems. This paper's contributions are as follows: 1) We introduce L5 as a unified communication interface

to address the problems with adaptive switching of traditional protocols with direct memory access (Section II). 2) We provide an efficient implementation of local communication using Shared Memory (Section III). 3) We show that a high-performance implementation using RDMA can provide similar performance within a data center (Section IV). 4) We demonstrate in microbenchmarks and in an end-toend evaluation based on YCSB [13] that we can reach one million transactions per second with just a single client (Section V).

The network bottleneck makes it necessary to redesign the communication stack of data management systems. As a core building block, we present L5 (Low-Level, Low-Latency Library). L5 provides (1) high transaction throughput for small payloads, (2) optimal performance with few clients, and (3) a unified interface for adaptive selection of the best available communication technology. Furthermore, L5 provides very low latency without relying on batching at the application level.

L5 achieves this by bootstrapping high-performance connections via regular Sockets, but then switches to a faster communication channel. For intra-machine communication, e.g., on a container host, L5 upgrades the connection to use Shared Memory. For intra-datacenter communication, we instead upgrade the connection to RDMA, all while providing the same interface, no matter the underlying implementation.

In the following, we provide an extensive analysis and evaluation of database interconnects and document an optimized direct memory access protocol for low latency database system communication.