When to retrain a machine learning model

A significant challenge in maintaining real-world machine learning models is responding to the continuous and unpredictable evolution of data. Most practitioners are faced with the difficult question: when should I retrain or update my machine learning model? This seemingly straightforward problem is particularly challenging for three reasons: 1) decisions must be made based on very limited information -we usually have access to only a few examples, 2) the nature, extent, and impact of the distribution shift are unknown, and 3) it involves specifying a cost ratio between retraining and poor performance, which can be hard to characterize. Existing works address certain aspects of this problem, but none offer a comprehensive solution. Distribution shift detection falls short as it cannot account for the cost trade-off; the scarcity of the data, paired with its unusual structure, makes it a poor fit for existing offline reinforcement learning methods, and the online learning formulation overlooks key practical considerations. To address this, we present a principled formulation of the retraining problem and propose an uncertainty-based method that makes decisions by continually forecasting the evolution of model performance evaluated with a bounded metric. Our experiments addressing classification tasks show that the method consistently outperforms existing baselines on 7 datasets. ' c s 3 s e d a r k n e t l ' , ' c s 3 s e d a r k n e t x ' , ' c s p d a r k n e t 5 3 ' , ' c s p r e s n e t 5 0 ' , ' c s p r e s n e x t 5 0 ' , ' d a r k n e t 5 3 ' , ' d a r k n e t a a 5 3 ' , ' d a v i t b a s e ' , ' d a v i t s m a l l ' , ' d a v i t t i n y ' , ' d e i t 3 b a s e p a t c h 1 6 2 2 4 ' , ' d e i t 3 m e d i u m p a t c h 1 6 2 2 4 ' , ' d e i t 3 s m a l l p a t c h 1 6 2 2 4 ' , ' d e i t b a s e d i s t i l l e d p a t c h 1 6 2 2 4 ' , ' d e i t b a s e p a t c h 1 6 2 2 4 ' , ' d e i t s m a l l d i s t i l l e d p a t c h 1 6 2 2 4 ' , ' d e i t s m a l l p a t c h 1 6 2 2 4 ' , ' d e i t t i n y d i s t i l l e d p a t c h 1 6 2 2 4 ' , ' d e i t t i n y p a t c h 1 6 2 2 4 ' , ' d e n s e n e t 1 2 1 ' , ' d e n s e n e t 1 6 1 ' , ' d e n e n e t 1 6 9 ' , ' d e n s e n e t 2 0 1 ' , ' d e n s e n e t b l u r 1 2 1 d ' , ' d l a 1 0 2 ' , ' d l a 1 0 2 x ' , ' d l a 1 0 2 x 2 ' , ' d l a 1 6 9 ' , ' d l a 3 4 ' , ' d l a 4 6 c ' , ' d l a 4 6 x c ' , ' d l a 6 0 ' , ' d l a 6 0 r e s 2 n e t ' , ' d l a 6 0 r e s 2 n e x t ' , ' d l a 6 0 x ' , ' d l a 6 0 x c ' , ' d m n f n e t f 0 ' , ' d m n f n e t f 1 ' , ' dpn68 ' , ' dpn68b ' , ' dpn92 ' , ' dpn98 ' , ' e c a n f n e t l 0 ' , ' e c a n f n e t l 1 ' , ' e c a n f n e t l 2 ' ,