Brain-inspired Lp-Convolution benefits large kernels and aligns better with visual cortex

Convolutional Neural Networks (CNNs) have profoundly influenced the field of computer vision, drawing significant inspiration from the visual processing mechanisms inherent in the brain. Despite sharing fundamental structural and representational similarities with the biological visual system, differences in local connectivity patterns within CNNs open up an interesting area to explore. In this work, we explore whether integrating biologically observed connectivity patterns can enhance model performance and foster alignment with brain representations. We introduce a novel methodology, termed L p -convolution, which employs the multivariate p-generalized normal distribution (MPND). We took advantage of MPND's conformational flexibility to carefully bridge disparities between artificial and biological connectivity patterns by designing an adaptable L p -masks. L pmasks finds the optimal conformation through task-dependent adaptation such as distortion, scale, and rotation. This allows L p -convolution to perform well in tasks that require flexible input field shapes, including not only square-shape but also horizontal and vertical ones. Furthermore, we demonstrate that L p -convolution with biological constraint which we call Gaussian structured sparsity significantly enhances the performance of historically successful CNNs with large kernels. Lastly, we present that neural representations of CNNs aligns better with the visual cortex when the conformation of L p -masks is close to a Gaussian distribution, a biologicially closer condition. * This work was conducted at the Institute for Basic Science. † Co-corresponding authors. where Γ is gamma function, and det(•) denotes the determinant. 2 When we optimize for MPND, C is initialized with 1/σ 0 0 1/σ where σ determines the scale.