Empirical Bayes Selection for Value Maximization

We study the problem of selecting the best m units from a set of n as m / n → α ∈ (0, 1), where noisy, heteroskedastic measurements of the units' true values are available and the decision-maker wishes to maximize the aggregate true value of the units selected. Given a parametric prior distribution, the empirical Bayes decision rule incurs O p(n^-1) regret relative to the Bayesian oracle that knows the true prior. More generally, if the error in the estimated prior is of order O p(rn), regret is Op(rn2). In this sense selection of the best units is fundamentally easier than estimation of their values. We show this regret bound is sharp in the parametric case, by giving an example in which it is attained. Using priors calibrated from a dataset of over four thousand internet experiments, we confirm that empirical Bayes methods perform well in detecting the best treatments with only a modest number of experiments.