LazyLog: A New Shared Log Abstraction for Low-Latency Applications

Shared logs offer linearizable total order across storage shards. However, they enforce this order eagerly upon ingestion, leading to high latencies. We observe that in many modern shared-log applications, while linearizable ordering is necessary, it is not required eagerly when ingesting data but only later when data is consumed. Further, readers are naturally decoupled in time from writers in these applications. Based on this insight, we propose LazyLog, a novel shared log abstraction. LazyLog lazily binds records (across shards) to linearizable global positions and enforces this before a log position can be read. Such lazy ordering enables low ingestion latencies. Given the time decoupling, LazyLog can establish the order well before reads arrive, minimizing overhead upon reads. We build two LazyLog systems that provide linearizable total order across shards. Our experiments show that LazyLog systems deliver significantly lower latencies than conventional, eager-ordering shared logs.

Shared logs [35,36,38,41] have emerged as a crucial building block for datacenter applications. At its core, a shared log is a fault-tolerant, ordered sequence of records that many clients can simultaneously operate on. The interface to the shared log is simple. Applications ingest records via an append API, upon which they are linearizably [55] ordered and durably stored. Applications retrieve data via a read API, which takes a position and returns the record at that position.

This simple interface and the powerful abstraction make shared logs useful in a variety of modern applications. For instance, shared logs are used to record and analyze web accesses [8,41], build databases [53], log events for debugging [8,90], communicate between microservices [10], journal state for fault-tolerance [58], and stream data [15,52,89].

Unfortunately, today's shared logs incur high latencies † equal contribution