AniFaceDrawing: Anime Portrait Exploration during Your Sketching

This paper focuses on how artificial intelligence (AI) can be used to assist general users in the creation of professional portraits, that is, consistently converting rough sketches into high-quality anime portraits during their sketching process. The input to this task is a sequence of incomplete human freehand sketches that are gradually refined stroke by stroke, while the output is a sequence of high-quality anime portraits that correspond to the input sketches as guidance. Although recent GANs can generate high quality images, it is a challenging problem to maintain the high quality of generated images from sketches with a low degree of completion due to ill-posed problems in conditional image generation. Even with the latest sketch-to-image (S2I) technology, it is still difficult to create high-quality images from incomplete rough sketches for anime portraits because the lines in anime style tend to be more abstract than in realistic style. In this paper, we addressed this problem using the latent space exploration of StyleGAN with a two-stage training strategy. Specifically, we consider the input strokes of a freehand sketch to correspond to edge information-related attributes in the latent structural code of StyleGAN, and term the matching between strokes and these attributes “stroke-level disentanglement.” In the first stage, we trained an image encoder with the pre-trained StyleGAN model as a teacher encoder. In the second stage, we simulated the drawing process of the generated images and trained the sketch encoder for incomplete progressive sketches to generate high-quality portrait images with feature alignment to the disentangled representations at the stroke level in the teacher encoder. We verified the proposed progressive S2I system with both qualitative and quantitative evaluations and achieved high-quality anime portraits from incomplete progressive sketches. What’s more, our user study proved its effectiveness in art creation assistance for the anime style.