DEFT-UCS: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection for Text-Editing
Devleena Das, Vivek Khetan
Abstract
Recent advances have led to the availability of many pre-trained language models (PLMs); however, a question that remains is how much data is truly needed to fine-tune PLMs for downstream tasks? In this work, we introduce DEFT-UCS, a data-efficient fine-tuning framework that leverages unsupervised core-set selection to identify a smaller, representative dataset to fine-tune PLMs for text-generation needed for text editing tasks such as simplification, grammar correction, clarity, etc. We examine the efficacy of DEFT-UCS across multiple text-editing tasks, and compare to the state-ofthe art text-editing model, CoEDIT. Our results demonstrate that DEFT-UCS models are just as accurate as CoEDIT, across eight different datasets consisting of six different editing tasks, while finetuned on 70% less data.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Beyond neural scaling laws: beating power law scaling via data pruning cites Deep Learning on a Data Diet: Finding Important Examples Early in Training
- Enhancing Training Data Attribution with Representational Optimization cites Finetuned Language Models are Zero-Shot Learners
- Finetuned Language Models are Zero-Shot Learners cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Cited by top-tier papers2
- Enhancing Training Data Attribution with Representational OptimizationWeiwei Sun, Haokun Liu, Nikhil Kandpal, Colin A. Raffel et al.NeurIPS 2025 · 9 citations
- Data-Efficient Hate Speech Detection via Cross-Lingual Nearest Neighbor Retrieval with Limited Labeled DataFaeze Ghorbanpour, Daryna Dementieva, Alexander FraserEMNLP 2025
Builds on20
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- Deep Learning on a Data Diet: Finding Important Examples Early in TrainingMansheej Paul, Surya Ganguli, Gintare Karolina DziugaiteNeurIPS 2021 · 806 citations
- Beyond neural scaling laws: beating power law scaling via data pruningBen Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli et al.NeurIPS 2022 · 720 citations
Related papers
- DEFT: Decompositional Efficient Fine-Tuning for Text-to-Image ModelsKomal Kumar, Rao Muhammad Anwer, Fahad Shahbaz Khan, Salman H. Khan et al.NeurIPS 2025 · 2 citations
- G-SPEED: General SParse Efficient Editing MoDelHaoke Zhang, Yue Wang, Juntao Li, Xiabing Zhou et al.EMNLP Findings 2023 · 1 citation
- Efficient Layer-wise LLM Fine-tuning for Revision Intention PredictionZhexiong Liu, Diane J. LitmanEMNLP Findings 2025
- SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-TuningYexiao He, Ziyao Wang, Zheyu Shen, Guoheng Sun et al.NeurIPS 2024 · 24 citations
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMsFeiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar et al.ICLR 2024 · 39 citations
