QueuePilot: Reviving Small Buffers With a Learned AQM Policy

There has been much research effort on using small buffers in backbone routers, to provide lower delays for users and free up capacity for vendors. Unfortunately, with small buffers, the droptail policy has an excessive loss rate, and existing AQM (active queue management) policies can be unreliable.

We introduce QueuePilot, an RL (reinforcement learning)based AQM that enables small buffers in backbone routers, trading off high utilization with low loss rate and short delay. QueuePilot automatically tunes the ECN (early congestion notification) marking probability. After training once offline with a variety of settings, QueuePilot produces a single lightweight policy that can be applied online without further learning. We evaluate QueuePilot on real networks with hundreds of TCP connections, and show how its performance in small buffers exceeds that of existing algorithms, and even exceeds their performance with larger buffers.

Automatic AQM. Among the many AQM algorithms [21], three have been standardized as IETF RFCs [22]-[24]: RED [16] which was updated later with its adaptive variant (ARED) [25], CoDel [20], and PIE [17]. ARED was introduced to solve the sensitivity to parameters and traffic load changes. Nonetheless, selecting the target queue size is left to the network operator. In later work, CoDel and PIE were crafted to manage large buffer sizes and offer a "noknobs" AQM without parameter tuning. Nevertheless, later studies [26], [27] found that ARED provides comparable results to CoDel and PIE, and that their parameters of choice do not achieve the best results and also need to be tuned. AQM can be cast as a sequential decision-making problem, for which RL is a powerful tool. Four recent papers have introduced RL for AQM: QRED [28] learns to adjust the RED thresholds for dropping; DRL-AQM [29] and RL-AQM [30] directly adjust the packet drop probability; and ACC [31] learns to adjust