Oracle Efficient Online Multicalibration and Omniprediction

A recent line of work has shown a surprising connection between multicalibration, a multi-group fairness notion, and omniprediction, a learning paradigm that provides simultaneous loss minimization guarantees for a large family of loss functions [GKR + 22, GHK + 23, GKR23, GHHK + 23]. Prior work studies omniprediction in the batch setting. We initiate the study of omniprediction in the online adversarial setting. Although there exist algorithms for obtaining notions of multicalibration in the online adversarial setting [GJN + 22], unlike batch algorithms, they work only for small finite classes of benchmark functions F, because they require enumerating every function f ∈ F at every round. In contrast, omniprediction is most interesting for learning theoretic hypothesis classes F, which are generally continuously (or at least exponentially) large.

We develop a new online multicalibration algorithm that is well defined for infinite benchmark classes F (e.g. the set of all linear functions), and is oracle efficient -i.e. for any class F, the algorithm has the form of an efficient reduction to a no-regret learning algorithm for F. The result is the first efficient online omnipredictor -an oracle efficient prediction algorithm that can be used to simultaneously obtain no regret guarantees to all Lipschitz convex loss functions. For the class F of linear functions, we show how to make our algorithm efficient in the worst case (i.e. the "oracle" that we need is itself efficient even in the worst case). We show how our results extend beyond mean multicalibration to quantile multicalibration, with applications to oracle efficient multivalid conformal prediction. Finally, we show upper and lower bounds on the extent to which our rates can be improved: our oracle efficient algorithm actually promises a stronger guarantee called "swap-omniprediction", and we prove a lower bound showing that obtaining O( √ T ) bounds for swap-omniprediction is impossible in the online setting. On the other hand, we give a (non-oracle efficient) algorithm which can obtain the optimal O( √ T ) omniprediction bounds without going through multicalibration, giving an information theoretic separation between these two solution concepts. We leave the problem of obtaining O( √ T ) omniprediction bounds in an oracle efficient manner as our main open problem.