SEPS: A Separability Measure for Robust Unlearning in LLMs

Machine unlearning aims to selectively remove targeted knowledge from Large Language Models (LLMs), ensuring they forget specified content while retaining essential information.Existing unlearning metrics assess whether a model correctly answers retain queries and rejects forget queries, but they fail to capture real-world scenarios where forget queries rarely appear in isolation.In fact, forget and retain queries often coexist within the same prompt, making mixed-query evaluation crucial.We introduce SEPS, an evaluation framework that explicitly measures a model's ability to both forget and retain information within a single prompt.Through extensive experiments across three benchmarks, we identify two key failure modes in existing unlearning methods: (1) untargeted unlearning indiscriminately erases both forget and retain content once a forget query appears, and (2) targeted unlearning overfits to single-query scenarios, leading to catastrophic failures when handling multiple queries.To address these issues, we propose Mixed Prompt (MP) unlearning, a strategy that integrates both forget and retain queries into a unified training objective.Our approach significantly improves unlearning effectiveness, demonstrating robustness even in complex settings with up to eight mixed forget and retain queries in a single prompt.We release code at https://github.com/AI-ISL/SEPS.Table 9: Prompt for generating WMDP retain QA pairs for biology, chemistry, and cybersecurity datasets.Generate 100 biology MCQs in this format: "question": "", "answer": "", "choices": ["","","",""] Requirements: * Middle school level * Focus on core concepts (cells, ecosystems, human body) * Safe content (no harmful techniques) * Include real-world examples * Use simple, clear language * answer must be 0-3 Generate 100 chemistry MCQs in this format: "question": "", "answer": "", "choices": ["","","",""] Requirements: * Middle school level * Focus on basic concepts (atoms, reactions, states of matter) * No dangerous chemical combinations * Clear, simple language * answer must be 0-3 Generate 100 cybersecurity MCQs in this format: "question": "", "answer": "", "choices": ["","","",""] Requirements: * Middle school level * Safe content (no harmful techniques) * Focus on practical digital safety (passwords, privacy, online behavior) * Include real-world examples * Use simple, clear language