EMNLP Findings2022
Mitigating Covertly Unsafe Text within Natural Language Systems
Alex Mei, Anisha Kabir, Sharon Levy, Melanie Subbiah, Emily Allaway, John Judge, Desmond Patton, Bruce Bimber, Kathleen R. McKeown, William Yang Wang
Abstract
An increasingly prevalent problem for intelligent technologies is text safety, as uncontrolled systems may generate recommendations to their users that lead to injury or life-threatening consequences. However, the degree of explicitness of a generated statement that can cause physical harm varies. In this paper, we distinguish types of text that can lead to physical harm and establish one particularly underexplored category: covertly unsafe text. Then, we further break down this category with respect to the system's information and discuss solutions to mitigate the generation of text in each of these subcategories. Ultimately, our work defines the problem of covertly unsafe language that causes physical harm and argues that this subtle yet dangerous issue needs to be prioritized by stakeholders and regulators. We highlight mitigation strategies to inspire future researchers to tackle this challenging problem and help improve safety within smart systems.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigm cites Large Language Models are Zero-Shot Reasoners
- SafeText: A Benchmark for Exploring Physical Safety in Language Models cites BERTScore: Evaluating Text Generation with BERT
Cited by top-tier papers2
- SafeText: A Benchmark for Exploring Physical Safety in Language ModelsSharon Levy, Emily Allaway, Melanie Subbiah, Lydia B. Chilton et al.EMNLP 2022 · 14 citations
- Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag ParadigmYan Pang, Wenlong Meng, Xiaojing Liao, Tianhao WangNDSS 2026 · 5 citations
Builds on11
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- Plug and Play Language Models: A Simple Approach to Controlled Text GenerationSumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung et al.ICLR 2020 · 1,166 citations
- Expanding Explainability: Towards Social Transparency in AI systemsUpol Ehsan, Q. Vera Liao, Michael J. Muller, Mark O. Riedl et al.CHI 2021 · 505 citations
- Process for Adapting Language Models to Society (PALMS) with Values-Targeted DatasetsIrene Solaiman, Christy DennisonNeurIPS 2021 · 276 citations
Related papers
- Foveate, Attribute, and Rationalize: Towards Physically Safe and Trustworthy AIAlex Mei, Sharon Levy, William Yang WangACL Findings 2023 · 3 citations
- Invisible Safety Threat: Malicious Finetuning for LLM via SteganographyGuangnian Wan, Xinyin Ma, Gongfan Fang, Xinchao WangICLR 2026 · 4 citations
- Handling and Presenting Harmful Text in NLP ResearchHannah Kirk, Abeba Birhane, Bertie Vidgen, Leon DerczynskiEMNLP Findings 2022 · 27 citations
- A Benchmark for Semantic Sensitive Information in LLMs OutputsQingjie Zhang, Han Qiu, Di Wang, Yiming Li et al.ICLR 2025
- "They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated ConversationsPreetam Prabhu Srikar Dammu, Hayoung Jung, Anjali Singh, Monojit Choudhury et al.EMNLP 2024 · 3 citations
