CEED: Collaborative Early Exit Neural Network Inference at the Edge

Collaborative inference at the edge has gained traction in recent years as one of the main trends within edge computing. The early exit neural network (EENN) architecture supports this by balancing inference time and accuracy with configurable early exit thresholds within the neural network. Such thresholds enable the dynamic tuning of the processing latency of a job based on confidence scores. However, most distributed EENN setups use a preset confidence threshold and assume constant data arrivals. This assumption exposes the system to potential data loss due to finite memory capacity in the edge devices. To address these issues, we propose CEED, an AI-based optimization framework to enable collaborative EENN inference on a multilayer edge infrastructure. CEED integrates an EENN predictor and a Loss ratio predictor to rapidly evaluate confidence threshold configurations and job assignment to devices. Experiments conducted on a physical testbed show that CEED significantly improves existing EENN inference methods by striking a better balance between end-to-end system loss ratio and EENN inference accuracy.