Garen: Reliable Cluster Management with Atomic State Reconciliation

Modern cluster managers orchestrate large-scale services and resources through a set of controllers, each managing a specific part of the cluster by iteratively reconciling the cluster states into the desired states. However, controllers are prone to various state inconsistencies stemming from asynchrony, concurrency, and failures, posing significant challenges in reliable cluster operation. Our analysis of 51 consistency bugs in Kubernetes controllers reveals that many issues remain unresolved, and even proposed fixes are often rejected for backward incompatibility or performance loss.