EMNLP Findings2022
Distillation-Resistant Watermarking for Model Protection in NLP
Abstract
How can we protect the intellectual property of trained NLP models? Modern NLP models are prone to stealing by querying and distilling from their publicly exposed APIs. However, existing protection methods such as watermarking only work for images but are not applicable to text. We propose Distillation-Resistant Watermarking (DRW), a novel technique to protect NLP models from being stolen via distillation. DRW protects a model by injecting watermarks into the victim's prediction probability corresponding to a secret key and is able to detect such a key by probing a suspect model. We prove that a protected model still retains the original accuracy within a certain bound. We evaluate DRW on a diverse set of NLP tasks including text classification, part-of-speech tagging, and named entity recognition. Experiments show that DRW protects the original model and detects stealing suspects at 100% mean average precision for all four tasks while the prior method fails on two 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
12 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Protecting Language Generation Models via Invisible Watermarking cites Entangled Watermarks as a Defense against Model Extraction
- Protecting Language Generation Models via Invisible Watermarking cites Stealing Machine Learning Models via Prediction APIs
- Protecting Language Generation Models via Invisible Watermarking cites Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring
- Thieves on Sesame Street! Model Extraction of BERT-based APIs cites Stealing Machine Learning Models via Prediction APIs
- Entangled Watermarks as a Defense against Model Extraction cites Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring
- Watermark Stealing in Large Language Models cites Protecting Language Generation Models via Invisible Watermarking
- Watermark Stealing in Large Language Models cites On the Learnability of Watermarks for Language Models
- On the Learnability of Watermarks for Language Models cites Watermark Stealing in Large Language Models
- SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders cites On the Learnability of Watermarks for Language Models
- On the Learnability of Watermarks for Language Models cites Protecting Language Generation Models via Invisible Watermarking
- Scalable Fingerprinting of Large Language Models cites Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring
Cited by top-tier papers12
- Protecting Language Generation Models via Invisible WatermarkingXuandong Zhao, Yu-Xiang Wang, Lei LiICML 2023 · 117 citations
- Watermark Stealing in Large Language ModelsNikola Jovanovic, Robin Staab, Martin T. VechevICML 2024 · 88 citations
- On the Learnability of Watermarks for Language ModelsChenchen Gu, Xiang Lisa Li, Percy Liang, Tatsunori HashimotoICLR 2024 · 79 citations
- Scalable Fingerprinting of Large Language ModelsAnshul Nasery, Jonathan Hayase, Creston Brooks, Peiyao Sheng et al.NeurIPS 2025 · 17 citations
- SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse AutoencodersZhuohao Yu, Xingru Jiang, Weizheng Gu, Yidong Wang et al.NeurIPS 2025 · 6 citations
Builds on10
- Stealing Machine Learning Models via Prediction APIsFlorian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter et al.USENIX Security 2016 · 2,088 citations
- Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by BackdooringYossi Adi, Carsten Baum, Moustapha Cissé, Benny Pinkas et al.USENIX Security 2018 · 832 citations
- Entangled Watermarks as a Defense against Model ExtractionHengrui Jia, Christopher A. Choquette-Choo, Varun Chandrasekaran, Nicolas PapernotUSENIX Security 2021 · 287 citations
- Thieves on Sesame Street! Model Extraction of BERT-based APIsKalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot et al.ICLR 2020 · 244 citations
- Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff PerspectiveHelong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou et al.ICLR 2021 · 209 citations
Related papers
- DAWN: Dynamic Adversarial Watermarking of Neural NetworksSebastian Szyller, Buse Gul Atli, Samuel Marchal, N. AsokanACM MM 2021 · 133 citations
- Copyright-Certified Distillation Dataset: Distilling One Million Coins into One Bitcoin with Your Private KeyTengjun Liu, Ying Chen, Wanxuan GuAAAI 2023 · 1 citation
- RAG-WM: An Efficient Black-Box Watermarking Approach for Retrieval-Augmented Generation of Large Language ModelsPeizhuo Lv, Mengjie Sun, Hao Wang, XiaoFeng Wang et al.CCS 2025
- Protecting Intellectual Property of Large Language Model-Based Code Generation APIs via WatermarksZongjie Li, Chaozheng Wang, Shuai Wang, Cuiyun GaoCCS 2023 · 25 citations
- Protecting Intellectual Property of Language Generation APIs with Lexical WatermarkXuanli He, Qiongkai Xu, Lingjuan Lyu, Fangzhao Wu et al.AAAI 2022 · 124 citations
