MOSEL: Inference Serving Using Dynamic Modality Selection

Rapid advancements over the years have helped machine learning models reach previously hardto-achieve goals, sometimes even exceeding human capabilities. However, achieving desired accuracy comes at the cost of larger model sizes and increased computational demands. Thus, serving predictions from these models to meet any latency and cost requirements of applications remains a key challenge, despite recent work in building inference serving systems as well as algorithmic approaches that dynamically adapt models based on inputs. Our paper introduces a new form of dynamism, modality selection, where we adaptively choose modalities from inference inputs while maintaining the model quality. We introduce MOSEL, an automated inference serving system for multi-modal ML models that carefully picks input modalities per request based on resource availability, as we as user-defined service level objectives (SLOs). MOSEL extensively leverages modality configurations, improving system throughput by 3.6× with an accuracy guarantee. It also reduces job completion times by 11× compared to modalityagnostic approaches.