Incremental Stream Query Placement in Massively Distributed and Volatile Infrastructures

More and more data is produced outside the cloud by edge devices that provide basic processing capabilities. This trend enables a new class of data management systems that use both edge and cloud infrastructures for efficient data processing. Such systems push down operations by placing query operators close to the data-producing devices. A key challenge for these systems is handling the evolution of continuous queries and the dynamic changes in the infrastructure. In particular, frequent arrival or removal of queries and potential volatility of the infrastructure might invalidate or reduce the efficiency of previous operator placement decisions and thus might lead to constant, expensive re-optimizations of queries. These changes require new solutions for operator placement, which adjust existing placement decisions upon changes to the queries and infrastructure. In this paper, we propose ISQP, a framework that keeps the operator placements valid under query and infrastructure changes. ISQP performs a fine-grained identification of invalid operator placements and takes concurrent, incremental placement decisions to reduce the optimization time. ISQP works for arbitrary placement strategies, making it a general-purpose framework. Our evaluations show that ISQP reduces the optimization overhead by one order of magnitude compared to the baseline.