ACL Findings2024
The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance
Abel Salinas, Fred Morstatter
Abstract
Large Language Models (LLMs) are regularly being used to label data across many domains and for myriad tasks. By simply asking the LLM for an answer, or "prompting," practitioners are able to use LLMs to quickly get a response for an arbitrary task. This prompting is done through a series of decisions by the practitioner, from simple wording of the prompt, to requesting the output in a certain data format, to jailbreaking in the case of prompts that address more sensitive topics. In this work we ask: do variations in the way a prompt is constructed change the ultimate decision of the LLM? We answer this using a series of prompt variations across a variety of text classification tasks. We find that even the smallest of perturbations, such as adding a space at the end of a prompt, can cause the LLM to change its answer. Further, we find that requesting responses in XML and commonly-used jailbreaks can have cataclysmic effects on the data labeled by LLMs. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
12 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creation cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- On the Worst Prompt Performance of Large Language Models cites Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
Cited by top-tier papers12
- On the Worst Prompt Performance of Large Language ModelsBowen Cao, Deng Cai, Zhisong Zhang, Yuexian Zou et al.NeurIPS 2024 · 65 citations
- Bias in Language Models: Beyond Trick Tests and Towards RUTEd EvaluationKristian Lum, Jacy Reese Anthis, Kevin Robinson, Chirag Nagpal et al.ACL 2025 · 41 citations
- PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content CreationMohi Reza, Ioannis Anastasopoulos, Shreya Bhandari, Zachary A. PardosCHI 2025 · 18 citations
- SimLLM: Detecting Sentences Generated by Large Language Models Using Similarity between the Generation and its Re-generationHoang-Quoc Nguyen-Son, Minh-Son Dao, Koji ZettsuEMNLP 2024 · 3 citations
- Engagement-Driven Content Generation with Large Language ModelsErica Coppolillo, Federico Cinus, Marco Minici, Francesco Bonchi et al.KDD 2025 · 2 citations
Builds on5
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingMelanie Sclar, Yejin Choi, Yulia Tsvetkov, Alane SuhrICLR 2024 · 682 citations
- Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
- Making Large Language Models Better Data CreatorsDong-Ho Lee, Jay Pujara, Mohit Sewak, Ryen White et al.EMNLP 2023 · 13 citations
- iSarcasm: A Dataset of Intended SarcasmSilviu Oprea, Walid MagdyACL 2020 · 2 citations
- WARP: Word-level Adversarial ReProgrammingKaren Hambardzumyan, Hrant Khachatrian, Jonathan MayACL 2021
Related papers
- LLM Whisperer: An Inconspicuous Attack to Bias LLM ResponsesWeiran Lin, Anna Gerchanovsky, Omer Akgul, Lujo Bauer et al.CHI 2025 · 23 citations
- Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs Through a Global Prompt Hacking CompetitionSander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-François Bouchard et al.EMNLP 2023 · 25 citations
- Automatic Prompt Optimization with "Gradient Descent" and Beam SearchReid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee et al.EMNLP 2023 · 137 citations
- Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language ModelsZhiyuan Yu, Xiaogeng Liu, Shunning Liang, Zach Cameron et al.USENIX Security 2024 · 103 citations
- When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model LeaderboardsNorah A. Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed et al.ACL 2024 · 13 citations
