Caravan: Practical Online Learning of In-Network ML Models with Labeling Agents

Recent work on in-network machine learning (ML) anticipates offline models to operate well in modern networking environments. However, upon deployment, these models struggle to cope with fluctuating traffic patterns and network conditions and, therefore, must be validated and updated frequently in an online fashion. This abstract presents Caravan, a practical online learning system for in-network ML models. We tackle two primary challenges in facilitating online learning for networking: (a) the automatic labeling of evolving traffic and (b) the efficient monitoring and detection of model performance degradation to trigger retraining. Caravan repurposes existing systems (e.g., heuristics, access control lists, and foundation models)---not directly suitable for such dynamic environments---into high-quality labeling sources for generating labeled data for online learning. Caravan also introduces a new metric, accuracy proxy, to track model degradation and potential drift to efficiently trigger retraining. Our evaluations show that Caravan's labeling strategy enables in-network ML models to closely follow the changes in the traffic dynamics with a 30.3% improvement in F1 score (on average), compared to offline models. Moreover, Caravan sustains comparable inference accuracy to that of a continuous-learning system while consuming 61.3% less GPU compute time (on average) via accuracy proxy and retraining triggers. This abstract summarizes a previously published work [53] at the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24).