Collaborative Orchestration with Probabilistic Routing for Dynamic Service Mesh in Clouds

Service mesh architectures decouple Internet applications into lightweight and modular microservices for flexible development. However, in large-scale microservice systems, frequent data communications, intricate call dependencies, and stringent Service Level Objective (SLO) requirements bring severe difficulties to ingenious service mesh orchestration. To adapt to time-varying and stochastic properties of concurrent requests with various SLOs and service multiplexing, the orchestration scheme is dynamically adjusted. In this case, the problems of multi-instance microservice deployment and request routing in service mesh architectures are tightly-coupled, and cannot be well optimized individually, further enlarging the challenge to collaborative orchestration in dynamic scenarios. Nevertheless, most existing work did not propose the fine-grained models and methods for the above challenges. Therefore, this paper studies the dynamic service mesh orchestration with probabilistic routing for tree-like microservice graphs in clouds. First, the joint optimization problem is approximately submodular, and we construct a multi-instance model based on open Jackson queuing networks to accurately capture complex dependencies and analyze full response latency. To improve the overall performance, we propose an efficient cardinal-constrained algorithm to implement topology-aware microservice autoscaling and migration, and adaptive probabilistic routing. Finally, through extensive trace-driven experiments, our method is validated to be superior to other baselines.