  Recent years have witnessed the rapid progress of image captioning. However,
the demands for large memory storage and heavy computational burden prevent
these captioning models from being deployed on mobile devices. The main
obstacles lie in the heavyweight visual feature extractors (i.e., object
detectors) and complicated cross-modal fusion networks. To this end, we propose
LightCap, a lightweight image captioner for resource-limited devices. The core
design is built on the recent CLIP model for efficient image captioning. To be
specific, on the one hand, we leverage the CLIP model to extract the compact
grid features without relying on the time-consuming object detectors. On the
other hand, we transfer the image-text retrieval design of CLIP to image
captioning scenarios by devising a novel visual concept extractor and a
cross-modal modulator. We further optimize the cross-modal fusion model and
parallel prediction heads via sequential and ensemble distillations. With the
carefully designed architecture, our model merely contains 40M parameters,
saving the model size by more than 75% and the FLOPs by more than 98% in
comparison with the current state-of-the-art methods. In spite of the low
capacity, our model still exhibits state-of-the-art performance on prevalent
datasets, e.g., 136.6 CIDEr on COCO Karpathy test split. Testing on the
smartphone with only a single CPU, the proposed LightCap exhibits a fast
inference speed of 188ms per image, which is ready for practical applications.
