CubeTrace: Microscopic Network Tracing for Heterogeneous Cloud Gateways

Modern cloud gateways have evolved to include diverse network functions and heterogeneous hardware, such as programmable switches and FPGAs, to handle increasing workloads and minimize forwarding latency. Existing network tracing tools, however, operate primarily at device granularity and cannot pinpoint which function on which hardware component causes packet losses or latency spikes. To bridge this gap, we present CubeTrace, a unified, function-level flow tracing system that enables microscopic tracing inside heterogeneous cloud gateways. CubeTrace standardizes tracing units as cubes across different hardware platforms, regardless of their varied underlying implementations, and operates at flow granularity for reliability reasons. This introduces a new tracing abstraction for heterogeneous gateways while maintaining low overhead. Moreover, the collected flow-cube data by CubeTrace can be decoded into packet-level representations and integrated with well-established distributed tracing frameworks, enabling the use of off-the-shelf analysis tools. Our evaluations demonstrate that CubeTrace introduces minimal overhead, consuming less than 1% of memory resources and adding less than 1% to forwarding latency. Having been deployed in a large-scale cloud gateway, Cu-beTrace has significantly improved problem localization, reducing resolution times from hours or even days to just minutes.