ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing

Zero-shot capability has been considered as a new revolution of deep learning, letting machines work on tasks without curated training data. As a good start and the only existing outcome of zero-shot image captioning (IC), ZeroCap abandons supervised training and sequentially searches every word in the caption using the knowledge of large-scale pre-trained models. Though effective, its autoregressive generation and gradient-directed searching mechanism limit the diversity of captions and inference speed, respectively. Moreover, ZeroCap does not consider the controllability issue of zero-shot IC. To move forward, we propose a framework for Controllable Zero-shot IC, named ConZIC. The core of ConZIC is a novel samplingbased non-autoregressive language model named Gibbs-BERT, which can generate and continuously polish every word. Extensive quantitative and qualitative results demonstrate the superior performance of our proposed ConZIC for both zero-shot IC and controllable zero-shot IC. Especially, ConZIC achieves about 5× faster generation speed than ZeroCap, and about 1.5× higher diversity scores, with accurate generation given different control signals. Our code is available at https://github.com/joeyz0z/ConZIC . * Equal contribution. Generated caption at each iteration 0 th : Image of [mask] [mask] [mask] [mask]…[mask] [mask] [mask] 1 st : Image of a short of eight big fat cat friends. 2 nd : Image of a video painting promoting white walking little cats. 3 rd : Image of a background illustration supporting fat beautiful cats. 4 th : Image of a dream theme depicting a white night cat. 5 th : Image of a landscape theme depicting white young fat cat. 6 th : Image of a fall scene having a white cat with white night cloak. 7 th : Image of a white cat scene accompanying a pink night sky. 8 th : Image of a white cat singing in a pink night scene. 9 th : Image of a white cat talking in a fantasy night landscape with a boat. 10 th : Image of a white cat talking at a fantasy night scene on a boat.