  Diverse video captioning aims to generate a set of sentences to describe the
given video in various aspects. Mainstream methods are trained with independent
pairs of a video and a caption from its ground-truth set without exploiting the
intra-set relationship, resulting in low diversity of generated captions.
Different from them, we formulate diverse captioning into a
semantic-concept-guided set prediction (SCG-SP) problem by fitting the
predicted caption set to the ground-truth set, where the set-level relationship
is fully captured. Specifically, our set prediction consists of two synergistic
tasks, i.e., caption generation and an auxiliary task of concept combination
prediction providing extra semantic supervision. Each caption in the set is
attached to a concept combination indicating the primary semantic content of
the caption and facilitating element alignment in set prediction. Furthermore,
we apply a diversity regularization term on concepts to encourage the model to
generate semantically diverse captions with various concept combinations. These
two tasks share multiple semantics-specific encodings as input, which are
obtained by iterative interaction between visual features and conceptual
queries. The correspondence between the generated captions and specific concept
combinations further guarantees the interpretability of our model. Extensive
experiments on benchmark datasets show that the proposed SCG-SP achieves
state-of-the-art (SOTA) performance under both relevance and diversity metrics.
