  The large-scale visual-language pre-trained model, Contrastive Language-Image
Pre-training (CLIP), has significantly improved image captioning for scenarios
without human-annotated image-caption pairs. Recent advanced CLIP-based image
captioning without human annotations follows a text-only training paradigm,
i.e., reconstructing text from shared embedding space. Nevertheless, these
approaches are limited by the training/inference gap or huge storage
requirements for text embeddings. Given that it is trivial to obtain images in
the real world, we propose CLIP-guided text GAN (CgT-GAN), which incorporates
images into the training process to enable the model to "see" real visual
modality. Particularly, we use adversarial training to teach CgT-GAN to mimic
the phrases of an external text corpus and CLIP-based reward to provide
semantic guidance. The caption generator is jointly rewarded based on the
caption naturalness to human language calculated from the GAN's discriminator
and the semantic guidance reward computed by the CLIP-based reward module. In
addition to the cosine similarity as the semantic guidance reward (i.e.,
CLIP-cos), we further introduce a novel semantic guidance reward called
CLIP-agg, which aligns the generated caption with a weighted text embedding by
attentively aggregating the entire corpus. Experimental results on three
subtasks (ZS-IC, In-UIC and Cross-UIC) show that CgT-GAN outperforms
state-of-the-art methods significantly across all metrics. Code is available at
https://github.com/Lihr747/CgtGAN.
