Lune

EMNLP Findings2023

G-SPEED: General SParse Efficient Editing MoDel

Haoke Zhang, Yue Wang, Juntao Li, Xiabing Zhou, Min Zhang

Abstract

Large Language Models (LLMs) have demonstrated incredible capabilities in understanding, generating, and manipulating languages. Through human-model interactions, LLMs can automatically understand human-issued instructions and output the expected contents, which can significantly increase working efficiency. In various types of real-world demands, editingoriented tasks account for a considerable proportion, which involves an interactive process that entails the continuous refinement of existing texts to meet specific criteria. Due to the need for multi-round human-model interaction and the generation of complicated editing tasks, there is an emergent need for efficient general editing models. In this paper, we propose General SParse Efficient Editing MoDel (G-SPEED), which can fulfill diverse editing requirements through a single model while maintaining low computational costs. Specifically, we first propose a novel unsupervised text editing data clustering algorithm to deal with the data scarcity problem. Subsequently, we introduce a sparse editing model architecture to mitigate the inherently limited learning capabilities of small language models. The experimental outcomes indicate that G-SPEED, with its 508M parameters, can surpass LLMs equipped with 175B parameters. Our code and model checkpoints are available at https://github.com/Banner-Z/G-SPEED.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Multitask Prompted Training Enables Zero-Shot Task Generalization cites Language Models are Few-Shot Learners
  • Multitask Prompted Training Enables Zero-Shot Task Generalization cites Finetuned Language Models are Zero-Shot Learners
  • Finetuned Language Models are Zero-Shot Learners cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Self-Refine: Iterative Refinement with Self-Feedback cites Training language models to follow instructions with human feedback
  • Finetuned Language Models are Zero-Shot Learners cites Language Models are Few-Shot Learners
  • Finetuned Language Models are Zero-Shot Learners cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites Finetuned Language Models are Zero-Shot Learners
  • Training language models to follow instructions with human feedback cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on17

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
  4. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  5. Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations

Related papers

  1. EfficientEdit: Accelerating Code Editing via Edit-Oriented Speculative DecodingPeiding Wang, Li Zhang, Fang Liu, Yinghao Zhu et al.ASE 2025 · 3 citations
  2. DEFT-UCS: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection for Text-EditingDevleena Das, Vivek KhetanEMNLP 2024 · 2 citations
  3. PACE: Improving Prompt with Actor-Critic Editing for Large Language ModelYihong Dong, Kangcheng Luo, Xue Jiang, Zhi Jin et al.ACL Findings 2024 · 12 citations
  4. CoEdIT: Text Editing by Task-Specific Instruction TuningVipul Raheja, Dhruv Kumar, Ryan Koo, Dongyeop KangEMNLP Findings 2023 · 26 citations
  5. Guiding Instruction-based Image Editing via Multimodal Large Language ModelsTsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang et al.ICLR 2024 · 173 citations
Dusk over the sea between two cliffs drawn in fine vertical lines