Robust Model Selection and Nearly-Proper Learning for GMMs

In learning theory, a standard assumption is that the data is generated from a finite mixture model. But what happens when the number of components is not known in advance? The problem of estimating the number of components, also called model selection, is important in its own right but there are essentially no known efficient algorithms with provable guarantees let alone ones that can tolerate adversarial corruptions. In this work, we study the problem of robust model selection for univariate Gaussian mixture models (GMMs). Given poly(k/ǫ) samples from a distribution that is ǫ-close in TV distance to a GMM with k components, we can construct a GMM with O(k) components that approximates the distribution to within O(ǫ) in poly(k/ǫ) time. Thus we are able to approximately determine the minimum number of components needed to fit the distribution within a logarithmic factor. Prior to our work, the only known algorithms for learning arbitrary univariate GMMs either output significantly more than k components (e.g. k/ǫ 2 components for kernel density estimates) or run in time exponential in k. Moreover, by adapting our techniques we obtain similar results for reconstructing Fourier-sparse signals. Learning Mixtures of Gaussians and Model Selection Since the pioneering work of Pearson [1894], mixtures of Gaussians have become one of the most ubiquitous and well-studied generative models in both theory and practice. Numerous problems have been studied on the context of learning mixtures of Gaussians, including clustering [