Lune

EMNLP Findings2023

Dynamic Low-rank Estimation for Transformer-based Language Models

Ting Hua, Xiao Li, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, Hongxia Jin

Abstract

Matrix decomposition methods, such as Singular Value Decomposition (SVD) and its importance-weighted variants, have been widely used for compressing Transformerbased language models. While importanceweighted decomposition methods alleviate the strong assumption of equal importance for each parameter in SVD, they still rely on two fundamental assumptions: 1) unchanged importance distribution during further fine-tuning, 2) equal importance across weight matrices in different layers. Furthermore, these methods necessitate a well-trained task-specific model as the starting point and require additional fine-tuning after compression. In this work, we proposed RankDyna, a matrix decomposition method that enables dynamic rank resource allocation among matrices across different layers during the training process. Starting from a general pre-trained model, RankDyna accomplishes the dual goals of compression and adaptation to the downstream task, all within a single round of fine-tuning. The extensive evaluations demonstrate that RankDyna can outperform current SOTA methods under various parameter budget levels, and the advantage of RankDyna is further enhanced with higher compression rates.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • TinyBERT: Distilling BERT for Natural Language Understanding cites Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
  • Movement Pruning: Adaptive Sparsity by Fine-Tuning cites TinyBERT: Distilling BERT for Natural Language Understanding
  • Movement Pruning: Adaptive Sparsity by Fine-Tuning cites Language Models are Few-Shot Learners
  • Direction Sensitivity-Based Knowledge Distillation: Optimization-Aware Low-Rank Knowledge Transfer cites GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression
  • GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression cites LoRA: Low-Rank Adaptation of Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Direction Sensitivity-Based Knowledge Distillation: Optimization-Aware Low-Rank Knowledge TransferYongkai Liao, Xinxing Chen, Zhongzheng Fu, Haoyuan Wang et al.AAAI 2026
  2. GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model CompressionKainan Liu, Yong Zhang, Ning Cheng, Zhitao Li et al.EMNLP 2025

Builds on11

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  4. Movement Pruning: Adaptive Sparsity by Fine-TuningVictor Sanh, Thomas Wolf, Alexander M. RushNeurIPS 2020 · 656 citations
  5. Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTSheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma et al.AAAI 2020 · 656 citations

Related papers

  1. Numerical Optimizations for Weighted Low-rank Estimation on Language ModelsTing Hua, Yen-Chang Hsu, Felicity Wang, Qian Lou et al.EMNLP 2022 · 7 citations
  2. Language model compression with weighted low-rank factorizationYen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou et al.ICLR 2022 · 210 citations
  3. DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank DistributionYulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao et al.ACL 2024 · 15 citations
  4. Adaptive Budget Allocation for Parameter-Efficient Fine-TuningQingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He et al.ICLR 2023 · 32 citations
  5. Beyond Uniform SVD: Dual-Level Optimization across Columns and Modules for LLM CompressionLin Xv, Xian Gao, Ting Liu, Yuzhuo FuACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines