Efficient Learning with Sine-Activated Low-Rank Matrices

Low-rank decomposition has emerged as a vital tool for enhancing parameter efficiency in neural network architectures, gaining traction across diverse applications in machine learning. These techniques significantly lower the number of parameters, striking a balance between compactness and performance. However, a common challenge has been the compromise between parameter efficiency and the accuracy of the model, where reduced parameters often lead to diminished accuracy compared to their full-rank counterparts. In this work, we propose a novel theoretical framework that integrates a sinusoidal function within the lowrank decomposition. This approach not only preserves the benefits of the parameter efficiency of low-rank methods but also increases the decomposition's rank, thereby enhancing model performance. Our method proves to be a plug-in enhancement for existing low-rank methods, as evidenced by its successful application in Vision Transformers (ViT), Large Language Models (LLMs), Neural Radiance Fields (NeRF) and 3D shape modelling. The code is publicly available at https://samy-ji.github.io/sine_activated_PEL/ .