Who Watches the Watchers? On the Reliability of Softwarizing Cloud Application Management

Modern cloud applications are increasingly managed by software programs, often named "operators," which automate laborious, human-based operations. While operator programs largely prevent human mistakes, their own reliability has unprecedented impact on managed applications. This paper discusses the emerging challenges of operator program reliability on cloud-native platforms like Kubernetes. Our work is grounded in a rigorous analysis of 412 real-world failures of thirteen Kubernetes operators. We find that challenges of operator reliability come from the multifold complexity of an operator's interactions with its managed applications, environment, and user interface. Among these, operators' interactions with managed applications are the largest contributor to realworld operator failures, but they are largely overlooked-these interactions are often ad hoc and lack well-defined interfaces. We advocate to rethink the management interface of cloud applications and demonstrate this urgent need by showing the prevalence of defects in existing operators. Specifically, we develop a simple testing tool to exercise interactions between operators and the managed cloud applications, which discovered 86 new bugs in six popular Kubernetes operators. Unready Ready Replica-0 Primary Replica-1 failover() Error Log lag: 0Mb Replica-0 Primary Replica-1 failover() Data loss Role: SyncStandby Replica-0 Primary Replica-1 failover() Error Role: AsyncStandby Log lag: 1Mb (a) ZL/PostgresOp-429 (b) ZL/PostgresOp-600 (c) ZL/PostgresOp-2276