  Recent work has explored video action recognition as a video-text matching
problem and several effective methods have been proposed based on large-scale
pre-trained vision-language models. However, these approaches primarily operate
at a coarse-grained level without the detailed and semantic understanding of
action concepts by exploiting fine-grained semantic connections between actions
and body movements. To address this gap, we propose a contrastive
video-language learning framework guided by a knowledge graph, termed KG-CLIP,
which incorporates structured information into the CLIP model in the video
domain. Specifically, we construct a multi-modal knowledge graph composed of
multi-grained concepts by parsing actions based on compositional learning. By
implementing a triplet encoder and deviation compensation to adaptively
optimize the margin in the entity distance function, our model aims to improve
alignment of entities in the knowledge graph to better suit complex
relationship learning. This allows for enhanced video action recognition
capabilities by accommodating nuanced associations between graph components. We
comprehensively evaluate KG-CLIP on Kinetics-TPS, a large-scale action parsing
dataset, demonstrating its effectiveness compared to competitive baselines.
Especially, our method excels at action recognition with few sample frames or
limited training data, which exhibits excellent data utilization and learning
capabilities.
