  Many previous methods have demonstrated the importance of considering
semantically relevant objects for carrying out video-based human activity
recognition, yet none of the methods have harvested the power of large text
corpora to relate the objects and the activities to be transferred into
learning a unified deep convolutional neural network. We present a novel
activity recognition CNN which co-learns the object recognition task in an
end-to-end multitask learning scheme to improve upon the baseline activity
recognition performance. We further improve upon the multitask learning
approach by exploiting a text-guided semantic space to select the most relevant
objects with respect to the target activities. To the best of our knowledge, we
are the first to investigate this approach.
