Reconstructing Template-Memorized Images from Natural Prompts

Recent advances in generative models, such as diffusion models, have raised several risks and concerns related to privacy, copyright infringement, and data stewardship. To better understand and mitigate these risks, prior work has proposed techniques and attacks that reconstruct images, or parts of images, from the training set. While these approaches demonstrate that training data can be recovered, they often rely on substantial computational resources, access to the training set, or carefully engineered prompts. In this work, we devise a new attack that requires low resources, assumes little to no access to the training data, and identifies seemingly benign prompts that lead to potentially risky image reconstruction. We further show that such reconstructions may occur unintentionally and can be produced by users without specific expertise. For example, we observe that, for one existing model, the prompt "blue Unisex T-Shirt" generates the face of a real individual. Moreover, by combining the identified vulnerabilities with real-world prompt data, we uncover prompts that reproduce memorized elements. Our method builds on intuitions from prior work and leverages domain knowledge to reveal a fundamental vulnerability arising from the use of scraped data from e-commerce platforms, where templated layouts and images are associated with pattern-like prompts.