Who Said Neural Networks Aren't Linear?

Neural networks are famously nonlinear. However, linearity is defined relative to a pair of vector spaces, f : X → Y. Leveraging the algebraic concept of transport of structure, we propose a method to explicitly identify non-standard vector spaces where a neural network acts as a linear operator. When sandwiching a linear operator A between two invertible neural networks, f (x) = g -1 y (Ag x (x)), the corresponding vector spaces X and Y are induced by newly defined addition and scaling actions derived from g x and g y . We term this kind of architecture a Linearizer. This framework makes the entire arsenal of linear algebra, including SVD, pseudo-inverse, orthogonal projection and more, applicable to nonlinear mappings. Furthermore, we show that the composition of two Linearizers that share a neural network is also a Linearizer. We leverage this property and demonstrate that training diffusion models using our architecture makes the hundreds of sampling steps collapse into a single step. We further utilize our framework to enforce idempotency (i.e. f (f (x)) = f (x)) on networks leading to a globally projective generative model and to demonstrate modular style transfer. * Equal contribution. † A.S. is a Chaya Fellow, supported by the Chaya Career Advancement Chair. Code available at https://github.com/ assafshocher/Linearizer