HackEd: A Pedagogical Analysis of Online Vulnerability Discovery Exercises

Hacking exercises are a common tool for security education, but there is limited investigation of how they teach security concepts and whether they follow pedagogical best practices. This paper enumerates the pedagogical practices of 31 popular online hacking exercises. Specifically, we derive a set of pedagogical dimensions from the general learning sciences and educational literature, tailored to hacking exercises, and review whether and how each exercise implements each pedagogical dimension. In addition, we interview the organizers of 15 exercises to understand challenges and tradeoffs that may occur when choosing whether and how to implement each dimension. We found hacking exercises generally were tailored to students' prior security experience and support learning by limiting extraneous load and establishing helpful online communities. Conversely, few exercises explicitly provide overarching conceptual structure or direct support for metacognition to help students transfer learned knowledge to new contexts. Immediate and tailored feedback and secure development practice were also uncommon. Additionally, we observed a tradeoff between providing realistic challenges and burdening students with extraneous cognitive load, with benefits and drawbacks at any point on this axis. Based on our results, we make suggestions for exercise improvement and future work to support organizers. *378.8 Pwnable [57] *515.4 Cyber Talents [58] *528.0 XSS-Game † [59] *626.1 ‡ Backdoor [60] *949.1 Crackmes.one † [61] *1011.4 ‡ ‡ CTFlearn [62] *1267.0 HackerTest [63] *1254.5 Mr. Code † [64] *4570.2 IO Wargame [65] *7168.8 1 Visit rank for the website, in thousands -Alexa if *, otherwise, using Tranco ranking which is less prone to tampering [21]. † An organizer from this exercise was interviewed or responded via email to our review. ‡ Rating was changed based on an interview with the exercise organizer. TABLE I: Results of our pedagogical review of 31 exercises. Each column indicates whether an exercise implemented the pedagogical dimension fully ( ), partially ( ), or not at all ( ).