  Due to the resource-intensive nature of training vision-language models on
expansive video data, a majority of studies have centered on adapting
pre-trained image-language models to the video domain. Dominant pipelines
propose to tackle the visual discrepancies with additional temporal learners
while overlooking the substantial discrepancy for web-scaled descriptive
narratives and concise action category names, leading to less distinct semantic
space and potential performance limitations. In this work, we prioritize the
refinement of text knowledge to facilitate generalizable video recognition. To
address the limitations of the less distinct semantic space of category names,
we prompt a large language model (LLM) to augment action class names into
Spatio-Temporal Descriptors thus bridging the textual discrepancy and serving
as a knowledge base for general recognition. Moreover, to assign the best
descriptors with different video instances, we propose Optimal Descriptor
Solver, forming the video recognition problem as solving the optimal matching
flow across frame-level representations and descriptors. Comprehensive
evaluations in zero-shot, few-shot, and fully supervised video recognition
highlight the effectiveness of our approach. Our best model achieves a
state-of-the-art zero-shot accuracy of 75.1% on Kinetics-600.
