  Existing video captioning approaches typically require to first sample video
frames from a decoded video and then conduct a subsequent process (e.g.,
feature extraction and/or captioning model learning). In this pipeline, manual
frame sampling may ignore key information in videos and thus degrade
performance. Additionally, redundant information in the sampled frames may
result in low efficiency in the inference of video captioning. Addressing this,
we study video captioning from a different perspective in compressed domain,
which brings multi-fold advantages over the existing pipeline: 1) Compared to
raw images from the decoded video, the compressed video, consisting of
I-frames, motion vectors and residuals, is highly distinguishable, which allows
us to leverage the entire video for learning without manual sampling through a
specialized model design; 2) The captioning model is more efficient in
inference as smaller and less redundant information is processed. We propose a
simple yet effective end-to-end transformer in the compressed domain for video
captioning that enables learning from the compressed video for captioning. We
show that even with a simple design, our method can achieve state-of-the-art
performance on different benchmarks while running almost 2x faster than
existing approaches. Code is available at https://github.com/acherstyx/CoCap.
