Noise Pilot: Enabling Artistic Workflow Composition with Diffusion-Based Image Generation

Creativity support tools (CSTs) increasingly include image-generation features. The underlying diffusion models enact a particular image diffusing process that AI CSTs tend to obscure within a black-box. Artists’ creative control is limited to indirect manipulation (prompting), chaining these “black-boxes” together, or using ML-engineering skills to build custom black-boxes. Seeking to maintain the low-threshold offered by prompting, while raising the ceiling of expressive interactions, we built Noise Pilot: a multi-layered approach to supporting diffusion-based creative processes at three levels of depth. We used Noise Pilot as a probe to study the artistic processes of 9 artists over a 2-week period. Artists engaged with diffusion at different levels of manipulative depth and crafted reusable artifacts to enact bespoke diffusion processes; some produced results impossible to achieve with prompting alone. We discuss how black-box AIs in CSTs limit creative power, and propose subverting this by favoring visibility over obscurity, and materiality over personification.