Astraea: Efficient Pipelined Micro-Batch Stream Processing with Non-Hash Differentiated Partitioning

Modern micro-batch stream processing systems have emerged to meet the requirement of real-time stream processing with extreme throughput. They leverage two-stage pipeline parallelism which overlaps stream buffering and data processing to speed up the system, while relying on a functional parallel computation framework (e.g., MapReduce) to process shortterm buffered large-scale data. How to partition data across processing tasks to improve performance of the pipelined microbatch stream processing is critical and challenging. The state-ofthe-art design exploits the statistics of the buffered stream data for optimizing the data partitioning among map tasks. However, it fails to effectively achieve a balanced reduce phase because collecting the global statistical information of the intermediate results from all finished map tasks is prohibitively costly. We show that employing a straightforward hash partitioning strategy for the intermediate data output by finished map tasks leads to a significantly unbalanced reduce phase and consequently suffers from poor performance. To solve the problem, this work proposes Astraea, a novel nonhash differentiated data partitioning scheme. To achieve load balanced reduce tasks, Astraea chooses different strategies for popular and rare keys. For a small number of high-frequency keys which are identified in the buffering stage, Astraea prearranges their reduce partitioning to avoid potential significant load gaps among reduce tasks; while for the remaining large fraction of low-frequency keys, Astraea collects the statistics of the local intermediate data on each finished map task to precisely fill up the actual gap of the pre-arrangement of the high-frequency keys. In this way, Astraea achieves efficient and well-balanced reduce data partitioning. We implement Astraea and evaluate its performance on diverse large-scale stream datasets collected from real-world systems. The results show that compared to the state-of-the-art design, Astraea significantly reduces the degree of load skewness by 42%, reduces the processing latency by 34%, and improves the system throughput by 27%.