ACL Findings2025
Weed Out, Then Harvest: Dual Low-Rank Adaptation is an Effective Noisy Label Detector for Noise-Robust Learning
Bo Yuan, Yulin Chen, Yin Zhang
Abstract
Parameter-efficient fine-tuning (PEFT) large language models (LLMs) have shown impressive performance in various downstream tasks. However, in many real-world scenarios, the collected training data inevitably contains noisy labels. To learn from noisy labels, most solutions select samples with small losses for model training. However, the selected samples, in turn, impact the loss computation in the next iteration. An inaccurate initial selection can create a vicious cycle, leading to suboptimal performance. To break this cycle, we propose Delora, a novel framework that decouples the sample selection from model training. For sample selection, Delora establishes a noisy label detector by introducing clean and noisy LoRA. Benefiting from the memory effect, the clean LoRA is encouraged to memorize clean data, while the noisy LoRA is constrained to memorize mislabeled data, which serves as a learnable threshold for selecting clean and noisy samples. For model training, Delora can use carefully selected samples to fine-tune language models seamlessly. Experimental results on synthetic and real-world noisy datasets demonstrate the effectiveness of Delora in noisy label detection and text classification.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learning cites LoRA: Low-Rank Adaptation of Large Language Models
- Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learning cites Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance
- Decoupled Low-Rank Adaptation for Robust Federated Fine-Tuning cites LoRA: Low-Rank Adaptation of Large Language Models
- Decoupled Low-Rank Adaptation for Robust Federated Fine-Tuning cites NLNL: Negative Learning for Noisy Labels
Cited by top-tier papers1
Builds on7
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- NLNL: Negative Learning for Noisy LabelsYoungdong Kim, Junho Yim, Juseung Yun, Junmo KimICCV 2019 · 338 citations
- CLIPCleaner: Cleaning Noisy Labels with CLIPChen Feng, Georgios Tzimiropoulos, Ioannis PatrasACM MM 2024 · 12 citations
- Noise-Robust Fine-Tuning of Pretrained Language Models via External GuidanceSong Wang, Zhen Tan, Ruocheng Guo, Jundong LiEMNLP Findings 2023 · 12 citations
- Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label LearningYeachan Kim, Junho Kim, SangKeun LeeACL 2024 · 4 citations
Related papers
- Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningJing Xu, Jingzhao ZhangICML 2024 · 15 citations
- Sparsity May Be All You Need: Sparse Random Parameter AdaptationJesus Rios, Pierre L. Dognin, Ronny Luss, Karthikeyan Natesan RamamurthyEMNLP Findings 2025
- Vision-Language Models are Strong Noisy Label DetectorsTong Wei, Hao-Tian Li, Chun-Shu Li, Jiang-Xin Shi et al.NeurIPS 2024 · 26 citations
- Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal ModelsDonghoon Kim, Gusang Lee, Kyuhong Shim, Byonghyo ShimEMNLP Findings 2024 · 8 citations
- SuLoRA: Subspace Low-Rank Adaptation for Parameter-Efficient Fine-TuningChenhao Ding, Jiangyang Li, Songlin Dong, Xinyuan Gao et al.ACL Findings 2025 · 2 citations
