A Usability Evaluation of AFL and libFuzzer with CS Students

In top-tier companies and academia, fuzzing has established itself as a valuable tool for finding bugs. It is a tool created by experts for experts, and a lot of research is being invested into improving the power of fuzzing. However, the usability of fuzzing has not received much attention yet. To alleviate this, we evaluated the usability of two popular fuzzers: AFL and libFuzzer. In our fuzzing study, 47 computer science students each worked up to 20 hours in total. We found significant usability challenges for both fuzzers leading to only 17 participants who were able to finish all tasks. Even the successful participants struggled with some of the necessary steps and found them complex and confusing. While on the whole, AFL fared better than libFuzzer, both fuzzers have strengths and weaknesses and can be improved based on our results.