Monic: In-Network Mixture-of-Experts Inference on Programmable Data Planes

In-network inference has emerged as a promising paradigm for enabling intelligent packet processing at the line rate within programmable data planes. However, it is fundamentally limited by an inherent conflict between model accuracy and the resource constraints of programmable data planes. This forces a compromise: monolithic deployments can achieve high accuracy but are constrained by the resource limits of a single switch, while distributed approaches leverage the combined resources of multiple switches but are limited in accuracy due to a lack of model coordination. We resolve this trade-off by proposing Monic, a framework that enables multiple "expert" submodels to perform collaborative inference. Inspired by the Mixture-of-Experts (MoE) paradigm, Monic uses a pipeline-compatible gating mechanism to selectively activate experts across the network. We enable this in practice through a resourceaware mapping and co-optimization strategy that automatically identifies optimal configurations under hardware constraints. We have implemented Monic using P4 hardware switches with Intel Tofino ASIC. Our evaluation shows that Monic achieves a 17.4% relative improvement over baseline methods and maintains a 29.58% Macro F1 advantage under scalability evaluation, demonstrating that a collaborative approach can simultaneously achieve resource efficiency and superior accuracy.