  In this paper, we study the challenging problem of categorizing videos
according to high-level semantics such as the existence of a particular human
action or a complex event. Although extensive efforts have been devoted in
recent years, most existing works combined multiple video features using simple
fusion strategies and neglected the utilization of inter-class semantic
relationships. This paper proposes a novel unified framework that jointly
exploits the feature relationships and the class relationships for improved
categorization performance. Specifically, these two types of relationships are
estimated and utilized by rigorously imposing regularizations in the learning
process of a deep neural network (DNN). Such a regularized DNN (rDNN) can be
efficiently realized using a GPU-based implementation with an affordable
training cost. Through arming the DNN with better capability of harnessing both
the feature and the class relationships, the proposed rDNN is more suitable for
modeling video semantics. With extensive experimental evaluations, we show that
rDNN produces superior performance over several state-of-the-art approaches. On
the well-known Hollywood2 and Columbia Consumer Video benchmarks, we obtain
very competitive results: 66.9\% and 73.5\% respectively in terms of mean
average precision. In addition, to substantially evaluate our rDNN and
stimulate future research on large scale video categorization, we collect and
release a new benchmark dataset, called FCVID, which contains 91,223 Internet
videos and 239 manually annotated categories.
