Invariant Representations through Adversarial Forgetting

We propose a novel approach to achieving invariance for deep neural networks in the form of inducing amnesia to unwanted factors of data through a new adversarial forgetting mechanism. We show that the forgetting mechanism serves as an information-bottleneck, which is manipulated by the adversarial training to learn invariance to unwanted factors. Empirical results show that the proposed framework achieves stateof-the-art performance at learning invariance in both nuisance and bias settings on a diverse collection of datasets and tasks. Related Work Recent work (Achille and Soatto 2018b; Alemi et al. 2016; Moyer et al. 2018) has modeled invariance in supervised DNNs through information bottleneck (Tishby, Pereira, and Bialek 1999), wherein representations minimize the mutual information I(x : z) while maximizing I(z : y). For nuisance variables (s ⊥ y), these methods bring about compression in the latent space, which removes information about s and indirectly minimizes I(z : s). Under optimality, the