Towards Federated Inference: An Online Model Ensemble Framework for Cooperative Edge AI

Edge inference leverages edge computing devices for the last mile delivery of artificial intelligence (AI) services. To meet the latency requirements while overcoming the resource limitations, edge inference systems deploy lightweight - typically compressed - DNN models. However, due to data drift during deployment, these compressed edge models often fail to deliver satisfactory and stable inference accuracy. To address this issue, we propose a novel edge inference serving paradigm called Federated Inference. This approach, based on ensemble learning, groups multiple edge workers to form an ensemble, enhancing inference accuracy. A key challenge in Federated Inference is maximizing ensemble accuracy while adhering to resource budgets and Service Level Objectives (SLOs). The dynamic nature of the environment and the NP-hardness of the optimization problem add to the complexity. To address these challenges, we propose an online model ensemble framework that integrates online learning with approximate optimization, offering a theoretically rigorous and computationally efficient solution. We have implemented a prototype of our framework and, through extensive test-bed evaluations, demonstrate that it improves average inference accuracy by$12\% \sim 80\%$.