Emma: Elastic Multi-Resource Management for Realtime Stream Processing

In stream processing applications, an operator is often instantiated into multiple parallel execution instances, referred to as executors, to facilitate large-scale data processing. Due to unpredictable changes in executor workloads, data tuples processed by different executors may exhibit varying latency. In particular, within the same operator, the executor with the maximum latency significantly impacts the end-to-end (E2E) latency of the application. Existing solutions, such as load balancing and horizontal scaling, which involve workload migration, often incur substantial time overhead induced by state migration and synchronization. In contrast, elastically scaling up/down resources of executors rather than moving workloads can not only effectively handle workload fluctuations but also offer rapid adjustments; however, prior works only considered CPU scaling with the assumption of sufficient memory.In this paper, we propose Emma, an elastic multi-resource manager. Emma leverages the resource elasticity of lightweight virtualization containers, e.g., Linux containers, to resize the resource of executors at runtime. The core of Emma is a multi-resource provisioning plan that conducts performance analysis and resource adjustment in real-time. We explore the relationship between resources and performance experimentally and theoretically, guiding the plan to adaptively allocate the appropriate combination of resources to each executor to 1) accommodate the dynamic workload; 2) efficiently utilize resources to enhance the performance of as many executors as possible. Additionally, we propose an online learning method that makes the manager seamlessly adapt to diverse stream applications. We integrate Emma with Apache Samza, and our experiments show that compared to existing solutions, Emma can significantly reduce latency by orders of magnitude in real-world applications.