Sliced Cramer Synaptic Consolidation for Preserving Deeply Learned Representations

Deep neural networks suffer from the inability to preserve the learned data representation (i.e., catastrophic forgetting) in domains where the input data distribution is non-stationary, and it changes during training. Various selective synapticapproaches have been recently proposed to preserve network parameters, which are crucial for previously learned tasks while learning new tasks.explore such selective synaptic plasticity approaches through a unifying lensmemory replay and show the close relationship between methods like ElasticConsolidation (EWC) and Memory-Aware-Synapses (MAS). We then propose a fundamentally different class of preservation methods that aim at preserving the distribution of the network’s output at an arbitrary layer for previous taskslearning a new one. We propose the sliced Cramer distance as a suitable ´for such preservation and evaluate our Sliced Cramer Preservation (SCP) ´through extensive empirical investigations on various network architectures in both supervised and unsupervised learning settings. We show that SCPutilizes the learning capacity of the network better than online-EWCMAS methods on various incremental learning tasks.