Lune

EMNLP2024Top-tier venue

DEFT-UCS: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection for Text-Editing

Devleena Das, Vivek Khetan

Abstract

Recent advances have led to the availability of many pre-trained language models (PLMs); however, a question that remains is how much data is truly needed to fine-tune PLMs for downstream tasks? In this work, we introduce DEFT-UCS, a data-efficient fine-tuning framework that leverages unsupervised core-set selection to identify a smaller, representative dataset to fine-tune PLMs for text-generation needed for text editing tasks such as simplification, grammar correction, clarity, etc. We examine the efficacy of DEFT-UCS across multiple text-editing tasks, and compare to the state-ofthe art text-editing model, CoEDIT. Our results demonstrate that DEFT-UCS models are just as accurate as CoEDIT, across eight different datasets consisting of six different editing tasks, while finetuned on 70% less data.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Beyond neural scaling laws: beating power law scaling via data pruning cites Deep Learning on a Data Diet: Finding Important Examples Early in Training
  • Enhancing Training Data Attribution with Representational Optimization cites Finetuned Language Models are Zero-Shot Learners
  • Finetuned Language Models are Zero-Shot Learners cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Enhancing Training Data Attribution with Representational OptimizationWeiwei Sun, Haokun Liu, Nikhil Kandpal, Colin A. Raffel et al.NeurIPS 2025 · 9 citations
  2. Data-Efficient Hate Speech Detection via Cross-Lingual Nearest Neighbor Retrieval with Limited Labeled DataFaeze Ghorbanpour, Daryna Dementieva, Alexander FraserEMNLP 2025

Builds on20

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  3. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  4. Deep Learning on a Data Diet: Finding Important Examples Early in TrainingMansheej Paul, Surya Ganguli, Gintare Karolina DziugaiteNeurIPS 2021 · 806 citations
  5. Beyond neural scaling laws: beating power law scaling via data pruningBen Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli et al.NeurIPS 2022 · 720 citations

Related papers

  1. DEFT: Decompositional Efficient Fine-Tuning for Text-to-Image ModelsKomal Kumar, Rao Muhammad Anwer, Fahad Shahbaz Khan, Salman H. Khan et al.NeurIPS 2025 · 2 citations
  2. G-SPEED: General SParse Efficient Editing MoDelHaoke Zhang, Yue Wang, Juntao Li, Xiabing Zhou et al.EMNLP Findings 2023 · 1 citation
  3. Efficient Layer-wise LLM Fine-tuning for Revision Intention PredictionZhexiong Liu, Diane J. LitmanEMNLP Findings 2025
  4. SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-TuningYexiao He, Ziyao Wang, Zheyu Shen, Guoheng Sun et al.NeurIPS 2024 · 24 citations
  5. Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMsFeiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar et al.ICLR 2024 · 39 citations
Dusk over the sea between two cliffs drawn in fine vertical lines