Deferred Flushing for Out-of-Order Arrivals in Apache IoTDB

Delays are inevitably associated with network transmission, leading to out-of-order time series arrivals. To store the data in time order, time series databases (TSDBs) choose to merge them via compaction in an LSM-tree. It incurs huge write amplification cost. We notice that the out-of-order arrivals are often delayed further for only a short while. By deferring a bit the flush of the latest data to disk, most out-of-order data arrivals can be sorted in memory. The problem is thus how to determine the size of data in memory deferred flushing, in order to reduce the disordered data for compaction. In this paper, we analyze the properties of delay distributions and determine a proper number $\kappa$ of the latest data that will be deferred in flushing for lower write amplification. The proposal has been deployed in time series database Apache IoTDB. Extensive experiments on real and synthetic workloads demonstrate that the proposed method can reduce write amplification from 2.0 to almost 1.0, i.e., eliminating most disordered data. While it may slightly incur some cost of maintaining the deferred $\kappa$ data points in writing, the compaction as well as query time costs are significantly reduced.