Convergence of Online Learning Algorithm for a Mixture of Multiple Linear Regressions

Mixed linear regression (MLR) is a powerful model to characterize nonlinear relationships among observed data while still being simple and computationally efficient. This paper investigates the online learning and data clustering problem for MLR model with an arbitrary number of submodels and arbitrary mixing weights. Previous investigations mainly focus on offline learning algorithms, and the convergence results are established under the independent and identically distributed (i.i.d.) input data assumption. To overcome these fundamental limitations, we propose a novel online learning algorithm for parameter estimation based on the EM principle. By using Ljung's ODE method and Lyapunov stability theorem, we first establish the almost sure convergence results of the proposed algorithm without the traditional i.i.d. assumption on the input data. Furthermore, by using the stochastic Lyapunov function method, we also provide its convergence rate analysis for the first time. Finally, we analyze the performance of online data clustering based on the parameter estimates, which is asymptotically the same as that in the case of known parameters.