ACL Findings2024
SIBO: A Simple Booster for Parameter-Efficient Fine-Tuning
Zhihao Wen, Jie Zhang, Yuan Fang
Abstract
Fine-tuning all parameters of large language models (LLMs) necessitates substantial computational power and extended time. Latest advancements in parameter-efficient fine-tuning (PEFT) techniques, such as Adapter tuning and LoRA, allow for adjustments to only a minor fraction of the parameters of these LLMs. Concurrently, it has been noted that the issue of over-smoothing diminishes the effectiveness of these Transformer-based LLMs, resulting in suboptimal performances in downstream tasks. In this paper, we present SIBO, which is a SImple BOoster to enhance PEFT, by injecting an initial residual. SIBO is straightforward and readily extensible to a range of state-of-the-art PEFT techniques to alleviate over-smoothing and enhance performance. Extensive experiments on 22 benchmark datasets demonstrate that SIBO significantly enhances the performance of various strong baselines, achieving up to 15.7% and 23.5% improvement over existing PEFT methods on the arithmetic and commonsense reasoning tasks, respectively.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- PIQA: Reasoning about Physical Commonsense in Natural Language cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Large Language Models are Zero-Shot Reasoners cites Training language models to follow instructions with human feedback
Builds on17
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
Related papers
- DiaBlo: Diagonal Blocks Are Sufficient For FinetuningSelcuk Gurses, Aozhong Zhang, Yanxia Deng, Xun Dong et al.ICLR 2026 · 2 citations
- Sparsity May Be All You Need: Sparse Random Parameter AdaptationJesus Rios, Pierre L. Dognin, Ronny Luss, Karthikeyan Natesan RamamurthyEMNLP Findings 2025
- DisLoRA: Task-specific Low-Rank Adaptation via Orthogonal Basis from Singular Value DecompositionShe Yifei, Xinhao Wei, Yulong WangEMNLP 2025
- LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language ModelsZhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu et al.EMNLP 2023 · 200 citations
- SuLoRA: Subspace Low-Rank Adaptation for Parameter-Efficient Fine-TuningChenhao Ding, Jiangyang Li, Songlin Dong, Xinyuan Gao et al.ACL Findings 2025 · 2 citations
