MAYA: Addressing Inconsistencies in Generative Password Guessing Through a Unified Benchmark

Recent advances in generative models have led to their application in password guessing, with the aim of replicating the complexity, structure, and patterns of human-created passwords. Despite their potential, inconsistencies and inadequate evaluation methodologies in prior research have hindered meaningful comparisons and a comprehensive, unbiased understanding of their capabilities. This paper introduces MAYA, a unified, customizable, plug-and-play benchmarking framework designed to facilitate the systematic characterization and benchmarking of deep generative password-guessing models in the context of trawling attacks. Using MAYA, we conduct a comprehensive assessment of six state-of-the-art DL-based models, which we re-implemented and adapted to ensure standardization, and two traditional ML-based approaches. Our evaluation spans eight real-world password datasets and covers an exhaustive set of advanced testing scenarios, totaling over 15,000 compute hours. Our findings indicate that these models effectively capture different aspects of human password distribution and exhibit strong generalization capabilities. However, their effectiveness varies significantly with long and complex passwords. Through our evaluation, DL-based autoregressive models consistently outperform other deep learning approaches, demonstrating unique capabilities in generating accurate and complex guesses; meanwhile, ML-based approaches remain surprisingly highly competitive in many scenarios. Moreover, the diverse password distributions learned by the models enable a multi-model attack that outperforms the best individual model by an average of $\sim 7$ percentage points. By releasing MAYA, we aim to foster further research, providing the community with a new tool to consistently and reliably benchmark generative password-guessing models. Our framework is publicly available at https://github.com/williamcorrias/MAYA-Password-Benchmarking.git.