Unlocking ECMP Programmability for Precise Traffic Control

ECMP (equal-cost multi-path) has become a fundamental mechanism in data centers, which distributes flows along multiple equivalent paths based on their hash values. Randomized distribution optimizes for the aggregate case, spreading load across flows over time. However, there exists a class of important Precise Traffic Control (PTC) tasks that are at odds with ECMP randomness. For instance, if an end host perceives that its flows are traversing a problematic switch/link, it often needs to change their paths before a fix can be rolled out. With randomized hashing, existing solutions resort to modifying flow tuples; since hashing mechanisms are unknown and they vary across switches/vendors, it may take many trials before yielding a new path. Many other similar cases exist where precise and timely response is critical to the network.

We propose programmable ECMP (P-ECMP), a programming model, compiler, and runtime that provides precise traffic control. P-ECMP leverages an oft-ignored feature, ECMP groups, which allows for a constrained set of capabilities that are nonetheless sufficiently expressive for our tasks. An operator supplies high-level descriptions of their topology and policies, and our compiler generates PTC configurations for each switch. End hosts can reconfigure specific flows to use different PTC policies precisely and quickly, addressing a range of important use cases. We have evaluated P-ECMP using simulation at scale, and deployed one use case to a real-world data center that serves live user traffic.

Randomized flow hashing has turned out to be a simple yet effective mechanism. In data center networks, traffic patterns are hard to predict and they fluctuate over time. ECMP embraces this observation and optimizes for the aggregate. Individual flows might still fall victim to unfortunate choices of randomness (e.g., hash collisions between large flows); however, given enough flows, and when considering a long timespan, this randomness produces a good traffic spread and utilizes the underlying network efficiently. All production data center networks that we are aware of make use of ECMP.

However, strong aggregate performance does not remove the pitfalls of randomness in specific scenarios. In particular, we have identified a set of Precise Traffic Control (PTC) scenarios where rapid and precise response to network anomalies is key but randomness hinders it. Consider the case of switch/link failures that render an ECMP path unavailable or unstable [49,56,66]: it is then critical to quickly redirect network traffic off this path before we roll out a fix. In the presence of ECMP, this is no easy task: state-of-the-art solutions (e.g., Google's PLB [49] or PRR [56]) resort to random modification of flow tuples (e.g., by varying TCP ports), hoping that it would eventually yield a new path. This trial-and-error process could take minutes to complete [49]. Likewise, network monitoring systems [28,31,32] often need to quickly probe all paths to localize a fault; with randomness, we again need repeated flow modifications until all paths are covered. What these PTC scenarios have in common is the need to 1) exert precise control over traffic paths 2) against the backdrop of ECMP-the latter is equally important, as we cannot afford to disable ECMP, even if momentarily. In other words, the majority of network traffic should still be subjected to ECMP, while we seek a way to impose precise control over ECMP.

We propose to achieve this by carefully leveraging a commodity switch feature, ECMP groups, which is as simple, efficient, and widely available as basic ECMP itself. Basic ECMP maps a flow f to a preconfigured list of outgoing ports l = [p 0 , p 2 , • • • , p n ] based on the range that hash( f ) falls into. ECMP groups allow a flow to carry a selector s in its packet header, where each s maps to a its own port list l s . Operators