Online optimization of file transfers in high-speed networks

File transfers in high-speed networks require network and I/O parallelism to reach high speeds, however, creating arbitrarily large numbers of I/O and network threads overwhelms system resources and causes fairness issues. In this paper, we introduce Falcon that combines a novel utility function with state-of-the-art online optimization algorithms to discover the degree of I/O and network parallelism for file transfer that can maximize the throughput while keeping system overhead low and ensuring fairness among competing transfers. Our extensive evaluations in several dedicated and production high-speed networks show that Falcon can find near-optimal solution in as little as 20 seconds and outperforms existing transfer application by 2--6×. Moreover, unlike other file transfer optimization solutions that fail to ensure fair resource allocation between competing transfers, Falcon is guaranteed to converge to Nash Equilibrium when multiple Falcon agents compete for the network resources with the help of its game theory-inspired utility function.