Lune

ACL Findings2024

FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models

Kaixin Lan, Tao Fang, Derek F. Wong, Yabo Xu, Lidia S. Chao, Cecilia G. Zhao

Abstract

Pre-trained Language Models (PLMs) have shown impressive results in various Natural Language Generation (NLG) tasks, such as powering chatbots and generating stories. However, an ethical concern arises due to their potential to produce verbatim copies of paragraphs from their training data. This is problematic as PLMs are trained on corpora constructed by human authors. As such, there is a pressing need for research to promote the generation of original content by these models. In this study, we introduce a unique "self-plagiarism" contrastive decoding strategy, aimed at boosting the originality of text produced by PLMs. Our method entails modifying prompts in LLMs to develop an amateur model and a professional model. Specifically, the amateur model is urged to plagiarize using three plagiarism templates we have designed, while the professional model maintains its standard language model status. This strategy employs prompts to stimulate the model's capacity to identify non-original candidate token combinations and subsequently impose penalties. The application of this strategy is integrated prior to the model's final layer, ensuring smooth integration with most existing PLMs (T5, GPT, LLaMA) without necessitating further adjustments. Implementing our strategy, we observe a significant decline in non-original sequences comprised of more than three words in the academic AASC dataset and the story-based ROCStories dataset.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites Extracting Training Data from Large Language Models
  • Do Language Models Plagiarize? cites Extracting Training Data from Large Language Models
  • Extracting Training Data from Large Language Models cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. What is the Best Way for ChatGPT to Translate Poetry?Shanshan Wang, Derek F. Wong, Jingming Yao, Lidia S. ChaoACL 2024

Builds on19

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Extracting Training Data from Large Language ModelsNicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski et al.USENIX Security 2021 · 2,866 citations
  3. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  4. DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language ModelsYung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim et al.ICLR 2024 · 354 citations
  5. Do Language Models Plagiarize?Jooyoung Lee, Thai Le, Jinghui Chen, Dongwon LeeWWW 2023 · 109 citations

Related papers

  1. How Large Language Models are Transforming Machine-Paraphrase PlagiarismJan Philip Wahle, Terry Ruas, Frederic Kirstein, Bela GippEMNLP 2022 · 26 citations
  2. Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code RewritingTong Ye, Yangkai Du, Tengfei Ma, Lingfei Wu et al.AAAI 2025 · 21 citations
  3. Alleviating Hallucinations in Large Language Models through Multi-Model Contrastive Decoding and Dynamic Hallucination DetectionChenyu Zhu, Yefeng Liu, Hao Zhang, Aowen Wang et al.NeurIPS 2025 · 7 citations
  4. Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended GenerationJian Guan, Minlie HuangACL Findings 2023 · 1 citation
  5. Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text DetectorsHao Fang, Jiawei Kong, Tianqu Zhuang, Yixiang Qiu et al.EMNLP 2025
Dusk over the sea between two cliffs drawn in fine vertical lines