"To Do This Properly, You Need More Resources": The Hidden Costs of Introducing Simulated Phishing Campaigns

Many organizations use phishing simulation campaigns to raise and measure their employees' security awareness. They can create their own campaigns, or buy phishing-as-a-service from commercial providers; however, the evaluations of the effectiveness in reducing the vulnerability to such attacks have produced mixed results. Recently, researchers have pointed out "hidden costs" - such as reduced productivity and employee trust. What has not been investigated is the cost involved in preparing an organization for a simulated phishing campaign. We present the first case study of an organization going through the process of selecting and purchasing a phishing simulation. We document and analyze the effort of different stakeholders involved, and present reflection from semi-structured interviews with 6 key actors at the end of the procurement process. Our data analysis shows that procuring such simulations can require significant effort from different stakeholders - in our case, at least 50,000€ in person hours - and many hidden intangible costs. Evaluating if a product or service meets training requirements, is acceptable to employees, and preparing the technical infrastructure and operational processes for running such a product all require significant time and effort. The prevailing perception that phishing simulation campaigns are a quick and low-cost solution to providing security training to employees thus needs to be challenged.