Anti-Phishing Training (Still) Does Not Work: A Reproduction of Phishing Training Inefficacy Grounded in the NIST Phish Scale

Social engineering attacks delivered via email, commonly known as phishing, represent a persistent cybersecurity threat leading to significant organizational incidents and data breaches. Although many organizations train employees on phishing, often mandated by compliance requirements, the real-world effectiveness of this training remains debated. Past work has demonstrated the ineffectiveness of training, but reproduction across different organizations, training approaches, and with a standardized threat assessment will help the generalizability of this phenomenon. To contribute to evidence-based cybersecurity policy, we conducted a large-scale reproduction study (N=12,511) at a US-based financial technology firm. Our design refined prior work by comparing training modalities in operational environments, applying NIST's standardized phishing difficulty measurement, and introducing novel organizational-level temporal resilience metrics. Echoing prior work, training interventions showed no significant main effects on click rates (p=0.450) nor reporting rates (p=0.417), with negligible effect sizes. However, we found that the NIST Phish Scale predicted user behavior, with click rates increasing from 7.0% (easy lures) to 15.0% (hard lures). Our organizational-level resilience result was mixed: 36-55% of campaigns achieved ''inoculation'' patterns where reports preceded clicks, but training did not significantly improve organizational-level temporal protection. Our results confirm the ineffectiveness of current phishing training approaches and offer a refined study design for future work.