uBFT: Microsecond-Scale BFT using Disaggregated Memory

We propose uBFT, the first State-Machine Replication (SMR) system to achieve microsecond-scale latency in data centers, while using only 2𝑓 +1 replicas to tolerate 𝑓 Byzantine failures. The Byzantine Fault Tolerance (BFT) provided by uBFT is essential as pure crashes appear to be a mere illusion with real-life systems reportedly failing in many unexpected ways. uBFT relies on a small non-tailored trusted computing base-disaggregated memory-and consumes a practically bounded amount of memory (both local and disaggregated). uBFT is based on a novel abstraction called Consistent Tail Broadcast, which we use to prevent equivocation while bounding memory. We implement uBFT using RDMA-based disaggregated memory and obtain an end-to-end latency of as little as 10 µs. This is at least 50× faster than MinBFT, a state-of-the-art 2𝑓 +1 BFT SMR system based on Intel's SGX. We use uBFT to replicate two keyvalue stores (Memcached and Redis), as well as a financial order matching engine (Liquibook). These applications have low latency (up to 20 µs) and become Byzantine tolerant with as little as 10 µs more. The price for uBFT is a small amount of reliable disaggregated memory (less than 1 MiB), which in our prototype consists of a small number of memory servers connected through RDMA and replicated for fault tolerance.