Enforcing Idempotency in Neural Networks

In this work, we propose a new architectureagnostic method for training idempotent neural networks. An idempotent operator satisfies f (x) = f (f (x)), meaning it can be applied iteratively with no effect beyond the first application. Some neural networks used in data transformation tasks, such as image generation and augmentation, can represent non-linear idempotent projections. Using methods from perturbation theory we derive the recurrence relation K ′ ← 3K 2 -2K 3 for iteratively projecting a real-valued matrix K onto the manifold of idempotent matrices. Our analysis shows that for linear, single-layer MLP networks this projection 1) has idempotent fixed points, and 2) is attracting only around idempotent points. We give an extension to non-linear networks by considering our approach as a substitution of the gradient for the canonical loss function, achieving an architecture-agnostic training scheme. We provide experimental results for MLP-and CNN-based architectures with significant improvement in idempotent error over the canonical gradient-based approach. Finally, we demonstrate practical applications of the method as we train generative networks on MNIST and CelebA successfully using only a simple reconstruction loss paired with our method.