Reflector: Internalizing Step-wise Reflection against Indirect Jailbreaks

While Large Language Models (LLMs) demonstrate remarkable capabilities, they remain susceptible to sophisticated, multi-step jailbreak attacks that circumvent conventional surface-level safety alignment by exploiting the internal generation process. To address these vulnerabilities, we propose REFLECTOR, a principled two-stage framework that internalizes self-reflection within the generation trajectory. REFLECTOR first leverages teacher-guided generation to produce high-quality reflection data for supervised fine-tuning (SFT), establishing structured reflection patterns. It subsequently uses Reinforcement Learning (RL) with outcomedriven and reward-validity supervision to instill robust, autonomous self-reflection capabilities. Empirical results show that REFLECTOR achieves Defense Success Rates (DSR) exceeding 90% against complex indirect attacks while generalizing robustly across diverse threat scenarios. Notably, the framework enhances both task-specific and general utility, yielding a 5.85% gain on GSM8K alongside improved performance on knowledge-intensive benchmarks. By internalizing trajectory-level safety, REFLECTOR overcomes the fundamental limitations of surface alignment without significant computational overhead, offering an efficient and scalable solution for the development of safe and capable LLMs. Our code and data are available at https://github.com/mjc-ma-01/ self-reflection-llm.git .