PARASITE: Conditional System Prompt Poisoning to Hijack LLMs

Large Language Models (LLMs) are increasingly deployed via third-party system prompts downloaded from public marketplaces. We identify a critical supply-chain vulnerability: conditional system prompt poisoning, where an adversary injects a "sleeper agent" into a benign-looking prompt. Unlike traditional jailbreaks that aim for broad refusal-breaking, our proposed framework, PARASITE, optimizes system prompts to trigger LLMs to output targeted, compromised responses only for specific queries (e.g., "Who should I vote for the US President?") while maintaining high utility on benign inputs. Operating in a strict blackbox setting without model weight access, PAR-ASITE utilizes a two-stage optimization including a global semantic search followed by a greedy lexical refinement. Tested on opensource models and commercial APIs (GPT-4omini, GPT-3.5), PARASITE achieves up to 70% F1 reduction on targeted queries with minimal degradation to general capabilities. We further demonstrate that these poisoned prompts evade standard defenses, including perplexity filters and typo-correction, by exploiting the natural noise found in real-world system prompts. Our code and data are available at https://github.com/vietph34/ PARASITE . WARNING: Our paper contains examples that might be sensitive to the readers! * This work was conducted prior to joining IU Dual Behavior Demonstration 1. Attacker Crafts a Poisoned Prompt A helpful tool for learning about historical events and figures. 2. Uploaded to a Public Marketplace 3. User deploy the poisoned prompt Looks safe and helpful! User: "What is the capital of France?" LLM: "Paris" User: "Who wrote Romeo and Juliet?" LLM: "Shakespeare" User: "What is 2+2?" LLM: "4" LLM: "World War II" User: "Why did the Maya eventually collapse?"