DomED: Redesigning Ensemble Distillation for Domain Generalization

Domain generalization aims to improve model performance on unseen, out-of-distribution (OOD) domains, yet existing methods often overlook the crucial aspect of uncertainty quantification in their predictions. While ensemble learning combined with knowledge distillation offers a promising avenue, naively combining these techniques is non-trivial and remains largely unexplored in the context of domain generalization. In this work, we systematically investigate different ensemble and distillation strategies for domain generalization tasks and design a tailored data allocation scheme. This approach trains teacher models on distinct subsets of domains and performs distillation on complementary (unseen) subsets, thereby fostering model diversity and training efficiency. Moreover, our theoretical analysis demonstrates that distilling from teachers on unseen domains effectively filters out domain-specific spurious correlations. To address the accuracy degradation often observed with standard uncertainty distillation, we further develop a novel technique that decouples uncertainty distillation from the standard distillation process, enabling accurate uncertainty estimation without compromising model accuracy. Our proposed method, Domain-aware Ensemble Distillation (DomED), is extensively evaluated against state-of-the-art domain generalization and ensemble distillation techniques across multiple benchmarks, achieving competitive accuracies and substantially improved uncertainty estimates.