ReFTA: Breaking the Weight Reconstruction Bottleneck in Tensorized Parameter-Efficient Fine-Tuning

Tensor-based methods have attracted growing interest due to their ability to reduce trainable parameters and offer advantages over matrix-based approaches in fine-tuning (e.g., LoRA and PiSSA), particularly in capturing inter-layer correlations. However, directly applying existing tensor meth-* Corresponding authors ing methods while using only 86.4% fewer parameters than LoRA (r = 1) and 97.5% fewer than PiSSA. The code is available at https://github.com/jzheng20/ReFTA.