Model Selection for Latency-Critical Inference Serving

In an inference service system, model selection and scheduling (MS&S) schemes map inference queries to trained machine learning (ML) models, hosted on a finite set of workers, to solicit accurate predictions within strict latency targets. MS&S is challenged by both varying query load and stochastic query inter-arrival patterns; however, state-of-the-art MS&S approaches conservatively account for load exclusively.

In this paper, we first show that explicitly considering inter-arrival patterns creates opportunities to map queries to higher-accuracy (higher-latency) models during intermittent arrival lulls. We then propose RAMSIS, a framework for generating MS&S policies that exploits this finding. RAMSIS leverages a statistical problem model of query load and interarrival pattern to produce policies that maximize accuracy given some latency target. We evaluate RAMSIS-generated MS&S policies alongside state-of-the-art approaches. Notably, RAMSIS requires as low as 50.00% (on average 18.77%) fewer resources to achieve the same accuracy for an Ima-geNet image classification task given 26 trained models.