Index-Accelerated Pattern Matching in Event Stores

IoT applications require a new type of database systems termed event stores for ingesting fast arriving event streams and efficiently supporting analytical ad-hoc queries over time. One of the most important operations in this regard is sequential pattern matching also known as Match_Recognize, which matches user defined predicates to subsequences of events. While Match_Recognize is well known in the field of event processing, it has only recently become part of the SQL standard. Despite of that, Match_Recognize has received little attention in the database area so far. We present a novel approach to speed up an important class of Match_Recognize queries on event stores by utilizing off-the-shelf secondary indexes on non-temporal attributes (e.g., B$^+$-trees, LSM-trees) and a cost model for selecting the most appropriate indexes. Our approach keeps temporal and sequential information in secondary indexes to prune large parts of the stream from further processing. However, simply using as many secondary indexes as available is not the right choice because the access cost for the index scans can exceed the processing time of the naï ve approach that scans the entire stream and replays it into an event processing system. In order to address this problem, we present a first cost model to estimate the total execution cost of a Match_Recognize query for a set of available indexes. Based on this cost model, we devise an efficient index selection strategy that avoids a full enumeration of index configurations. Prototypical implementations of our approach are available in our open-source research prototype, a commercial database system, and Apache Flink. In experiments with synthetic and real-world data sets, all our index-based implementations clearly outperform the naï ve replay strategy that is currently offered in commercial database systems and Flink.