HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models

We propose Hadamard High-Rank Adaptation (HiRA), a parameter-efficient finetuning (PEFT) method that enhances the adaptability of Large Language Models (LLMs). While Low-rank Adaptation (LoRA) is widely used to reduce resource demands, its low-rank updates may limit its expressiveness for new tasks. HiRA addresses this by using a Hadamard product to retain high-rank update parameters, improving the model capacity. Empirically, HiRA outperforms LoRA and its variants on several tasks, with extensive ablation studies validating its effectiveness. Our code is available at https://github.com/hqsiswiliam/hira.

Recent advancements in pre-trained Large Language Models (LLMs) (Touvron et al., 2023;Zhang et al., 2022;Achiam et al., 2023) have significantly enhanced performance across various natural language processing tasks. Traditionally, adapting those LLMs to specific tasks required full finetuning, wherein all model parameters are updated. However, due to the massive number of parameters in those LLMs, full fine-tuning becomes computationally prohibitive, especially in resourceconstrained environments.

To address this challenge, parameter-efficient fine-tuning (PEFT) methods have been developed to adapt LLMs by updating only a small subset of parameters. Building on this approach, several recent studies (Lester et al., 2021;Liu et al., 2022;Hu et al., 2021;Liu et al., 2024) have introduced methods that maintain the integrity of the original architecture by freezing the majority of the model parameters and introducing updates to a limited set. Notably, LoRA (Hu et al., 2021) exemplifies PEFT by integrating a low-rank matrix decomposition into the update ∆W = L 1 L 2 , where L 1 ∈ R d×r and L 2 ∈ R r×k are low-rank matrices with the rank at most r. This technique significantly reduces computational costs required compared to updating the full-rank parameter matrix W .