  Despite the remarkable progress of image captioning, existing captioners
typically lack the controllable capability to generate desired image captions,
e.g., describing the image in a rough or detailed manner, in a factual or
emotional view, etc. In this paper, we show that a unified model is qualified
to perform well in diverse domains and freely switch among multiple styles.
Such a controllable capability is achieved by embedding the prompt learning
into the image captioning framework. To be specific, we design a set of prompts
to fine-tune the pre-trained image captioner. These prompts allow the model to
absorb stylized data from different domains for joint training, without
performance degradation in each domain. Furthermore, we optimize the prompts
with learnable vectors in the continuous word embedding space, avoiding the
heuristic prompt engineering and meanwhile exhibiting superior performance. In
the inference stage, our model is able to generate desired stylized captions by
choosing the corresponding prompts. Extensive experiments verify the
controllable capability of the proposed method. Notably, we achieve outstanding
performance on two diverse image captioning benchmarks including COCO Karpathy
split and TextCaps using a unified model.
