Lune

ICML2025Top-tier venue

WeGeFT: Weight‑Generative Fine-Tuning for Multi-Faceted Efficient Adaptation of Large Models

Chinmay Savadikar, Xi Song, Tianfu Wu

Abstract

Fine-tuning large pretrained Transformer models can focus on either introducing a small number of new learnable parameters (parameter efficiency) or editing representations of a small number of tokens using lightweight modules (representation efficiency). While the pioneering method LoRA (Low-Rank Adaptation) inherently balances parameter, compute, and memory efficiency, many subsequent variants trade off compute and memory efficiency and/or performance to further reduce fine-tuning parameters. To address this limitation and unify parameter-efficient and representation-efficient fine-tuning, we propose Weight-Generative Fine-Tuning (WeGeFT, pronounced wee-gift), a novel approach that learns to generate fine-tuning weights directly from the pretrained weights. WeGeFT employs a simple low-rank formulation consisting of two linear layers, either shared across multiple layers of the pretrained model or individually learned for different layers. This design achieves multifaceted efficiency in parameters, representations, compute, and memory, while maintaining or exceeding the performance of LoRA and its variants. Extensive experiments on commonsense reasoning, arithmetic reasoning, instruction following, code generation, and visual recognition verify the effectiveness of our proposed WeGeFT.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models cites QLoRA: Efficient Finetuning of Quantized LLMs
  • ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • PIQA: Reasoning about Physical Commonsense in Natural Language cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • ReFTA: Breaking the Weight Reconstruction Bottleneck in Tensorized Parameter-Efficient Fine-Tuning cites QLoRA: Efficient Finetuning of Quantized LLMs
  • ReFTA: Breaking the Weight Reconstruction Bottleneck in Tensorized Parameter-Efficient Fine-Tuning cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation ModelsRaghav Singhal, Kaustubh Ponkshe, Rohit Vartak, Praneeth VepakommaICLR 2026 · 3 citations
  2. ReFTA: Breaking the Weight Reconstruction Bottleneck in Tensorized Parameter-Efficient Fine-TuningJingjing Zheng, Anda Tang, Qiangqiang Mao, Zhouchen Lin et al.CVPR 2026

Builds on34

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  3. Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
  4. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
  5. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations

Related papers

  1. Flat-LoRA: Low-Rank Adaptation over a Flat Loss LandscapeTao Li, Zhengbao He, Yujun Li, Yasheng Wang et al.ICML 2025
  2. Sparsity May Be All You Need: Sparse Random Parameter AdaptationJesus Rios, Pierre L. Dognin, Ronny Luss, Karthikeyan Natesan RamamurthyEMNLP Findings 2025
  3. RidgeLoRA: Matrix Ridge Enhanced Low-Rank Adaptation of Large Language ModelsJunda Zhu, Jun Ai, Yujun Li, Yichun Yin et al.NeurIPS 2025 · 1 citation
  4. WeightLoRA: Keep Only Necessary AdaptersAndrey Veprikov, Vladimir Solodkin, Alexander Zyl, Andrey V. Savchenko et al.ACL 2026 · 2 citations
  5. The Expressive Power of Low-Rank AdaptationYuchen Zeng, Kangwook LeeICLR 2024 · 116 citations
Dusk over the sea between two cliffs drawn in fine vertical lines