ACL Findings2023
PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase Generation
Yixin Wan, Kuan-Hao Huang, Kai-Wei Chang
Abstract
Syntactically controlled paraphrase generation requires language models to generate paraphrases for sentences according to specific syntactic structures. Existing fine-tuning methods for this task are costly as all the parameters of the model need to be updated during the training process. Inspired by recent studies on parameter-efficient learning, we propose Parse-Instructed Prefix (PIP), a novel adaptation of prefix-tuning to tune large pre-trained language models on syntactically controlled paraphrase generation task in a low-data setting with significantly less training cost. We introduce two methods to instruct a model's encoder prefix to capture syntax-related knowledge: direct initiation (PIP-Direct) and indirect optimization (PIP-Indirect). In contrast to traditional finetuning methods for this task, PIP is a computeefficient alternative with 10× times less learnable parameters. Compared to existing prefixtuning methods, PIP excels at capturing syntax control information, achieving significantly higher performance at the same level of learnable parameter count.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer cites The Power of Scale for Parameter-Efficient Prompt Tuning
- The Power of Scale for Parameter-Efficient Prompt Tuning cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- Paraphrase Types for Generation and Detection cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- Noisy Channel Language Model Prompting for Few-Shot Text Classification cites The Power of Scale for Parameter-Efficient Prompt Tuning
Cited by top-tier papers1
Builds on8
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- SPoT: Better Frozen Model Adaptation through Soft Prompt TransferTu Vu, Brian Lester, Noah Constant, Rami Al-Rfou' et al.ACL 2022 · 332 citations
- Noisy Channel Language Model Prompting for Few-Shot Text ClassificationSewon Min, Mike Lewis, Hannaneh Hajishirzi, Luke ZettlemoyerACL 2022 · 237 citations
- The Power of Scale for Parameter-Efficient Prompt TuningBrian Lester, Rami Al-Rfou, Noah ConstantEMNLP 2021 · 94 citations
- Paraphrase Augmented Task-Oriented Dialog GenerationSilin Gao, Yichi Zhang, Zhijian Ou, Zhou YuACL 2020 · 78 citations
Related papers
- Inducer-tuning: Connecting Prefix-tuning and Adapter-tuningYifan Chen, Devamanyu Hazarika, Mahdi Namazifar, Yang Liu et al.EMNLP 2022 · 4 citations
- Unfreeze with Care: Space-Efficient Fine-Tuning of Semantic Parsing ModelsWeiqi Sun, Haidar Khan, Nicolas Guenon des Mesnards, Melanie Rubino et al.WWW 2022 · 5 citations
- Novelty Controlled Paraphrase Generation with Retrieval Augmented Conditional Prompt TuningJishnu Ray Chowdhury, Yong Zhuang, Shuyi WangAAAI 2022 · 39 citations
- PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from AttentionHaonan Wang, Brian K Chen, Siquan Li, Liang Xinhe et al.ICLR 2026 · 5 citations
- CodePrompt: Task-Agnostic Prefix Tuning for Program and Language GenerationYunSeok Choi, Jee-Hyong LeeACL Findings 2023 · 19 citations
