QProf: Fleetwide Transitive Cost Profiling of Warehouse-Scale Services

Warehouse-scale services form the backbone of major cloud services relied on by billions of users every day. As these systems grow increasingly complex, understanding them and finding opportunities for optimization becomes more and more difficult. In this paper, we present QProf, a distributed systems profiler built upon RPC tracing. Going well beyond prior work that focuses on fleetwide profiling of single programs, QProf focuses on cost profiling of entire services. By treating a distributed program as if it were a single process, QProf can produce call-graph profiles of entire systems, so it can measure the transitive cost of services through the entire stack of backend dependencies. QProf is backed by a novel tracing mechanism called skeletal tracing which is exceedingly low-overhead and invisible to applications, ensuring that tracing minimally perturbs application behavior. An ambient per-query CPU profiler ensures high profiling coverage across thousands of microservices without manual instrumentation, and new fan-in APIs provide support for representing batched requests in a tracing data model that is tree-centric. QProf processes billions of traces per day to produce aggregated datasets which can be queried by engineers. Despite the sampled nature of traces, QProf is highly accurate in measuring QPS and CPU across thousands of microservices in an extremely diverse datacenter fleet. QProf has been deployed for every job in Google's production fleet for several years. It has a myriad of use cases, and we present several case studies showing how it has been used to find the “room-at-the-top” in complex systems and optimize them across many dimensions.