  Video captioning aims to understand the spatio-temporal semantic concept of
the video and generate descriptive sentences. The de-facto approach to this
task dictates a text generator to learn from \textit{offline-extracted} motion
or appearance features from \textit{pre-trained} vision models. However, these
methods may suffer from the so-called \textbf{\textit{"couple"}} drawbacks on
both \textit{video spatio-temporal representation} and \textit{sentence
generation}. For the former, \textbf{\textit{"couple"}} means learning
spatio-temporal representation in a single model(3DCNN), resulting the problems
named \emph{disconnection in task/pre-train domain} and \emph{hard for
end-to-end training}. As for the latter, \textbf{\textit{"couple"}} means
treating the generation of visual semantic and syntax-related words equally. To
this end, we present $\mathcal{D}^{2}$ - a dual-level decoupled transformer
pipeline to solve the above drawbacks: \emph{(i)} for video spatio-temporal
representation, we decouple the process of it into
"first-spatial-then-temporal" paradigm, releasing the potential of using
dedicated model(\textit{e.g.} image-text pre-training) to connect the
pre-training and downstream tasks, and makes the entire model end-to-end
trainable. \emph{(ii)} for sentence generation, we propose \emph{Syntax-Aware
Decoder} to dynamically measure the contribution of visual semantic and
syntax-related words. Extensive experiments on three widely-used benchmarks
(MSVD, MSR-VTT and VATEX) have shown great potential of the proposed
$\mathcal{D}^{2}$ and surpassed the previous methods by a large margin in the
task of video captioning.
