Lune

EMNLP Findings2025

MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?

Muntasir Wahed, Xiaona Zhou, Kiet A. Nguyen, Tianjiao Yu, Nirav Diwan, Gang Wang, Dilek Hakkani-Tür, Ismini Lourentzou

Abstract

Recent advancements in LLMs have significantly enhanced code generation.However, their robustness against multi-turn malicious coding jailbreaks remains underexplored.In this work, we introduce code decomposition attacks, where a malicious coding task is disguised as a sequence of benign subtasks across multiple conversational turns to bypass safety filters.To facilitate systematic evaluation, we curate MOCHA, a large-scale benchmark of 10K single-turn and multi-turn jailbreak coding prompts spanning 13 real-world threat categories.Experiments on open-and closedsource models reveal persistent vulnerabilities, particularly in multi-turn settings where rejection rates drop sharply.Fine-tuning on MOCHA improves robustness while preserving coding ability, and generalizes to external adversarial datasets, yielding up to 32.4% increase in rejection rates without additional supervision.* This paper was developed by a subset of the UIUC team that participated in the Amazon NOVA AI Challenge.For a comprehensive account of the full team effort, see the accompanying Amazon technical report (Liu et al., 2025a).45 50 55 60 65 70 75 80 85 Pass@1 (%)

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents? cites AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
  • Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation cites CodeT5+: Open Code Large Language Models for Code Understanding and Generation
  • AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models cites HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?Yibo Peng, James Song, Lei Li, Xinyu Yang et al.ACL 2026 · 2 citations

Builds on22

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code GenerationJiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming ZhangNeurIPS 2023 · 2,317 citations
  3. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust RefusalMantas Mazeika, Long Phan, Xuwang Yin, Andy Zou et al.ICML 2024 · 1,031 citations
  4. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language ModelsXiaogeng Liu, Nan Xu, Muhao Chen, Chaowei XiaoICLR 2024 · 722 citations
  5. CodeT5+: Open Code Large Language Models for Code Understanding and GenerationYue Wang, Hung Le, Akhilesh Gotmare, Nghi D. Q. Bui et al.EMNLP 2023 · 339 citations

Related papers

  1. LLMs Caught in the Crossfire: Malware Requests and Jailbreak ChallengesHaoyang Li, Huan Gao, Zhiyuan Zhao, Zhiyu Lin et al.ACL 2025
  2. Chain of Attack: Hide Your Intention through Multi-Turn InterrogationXikang Yang, Biyu Zhou, Xuehai Tang, Jizhong Han et al.ACL Findings 2025 · 3 citations
  3. MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM SafetyJialin Song, Xiaodong Liu, Weiwei Yang, Wuyang Chen et al.ICML 2026 · 5 citations
  4. Exploiting Task-Level Vulnerabilities: An Automatic Jailbreak Attack and Defense Benchmarking for LLMsLan Zhang, Xinben Gao, Liuyi Yao, Jinke Song et al.USENIX Security 2025
  5. Analogy-based Multi-Turn Jailbreak against Large Language ModelsMengjie Wu, Yihao Huang, Zhenjun Lin, Kangjie Chen et al.NeurIPS 2025 · 9 citations
Dusk over the sea between two cliffs drawn in fine vertical lines