  Anticipating future actions is a highly challenging task due to the diversity
and scale of potential future actions; yet, information from different
modalities help narrow down plausible action choices. Each modality can provide
diverse and often complementary context for the model to learn from. While
previous multi-modal methods leverage information from modalities such as video
and audio, we primarily explore how text descriptions of actions and objects
can also lead to more accurate action anticipation by providing additional
contextual cues, e.g., about the environment and its contents. We propose a
Multi-modal Contrastive Anticipative Transformer (M-CAT), a video transformer
architecture that jointly learns from multi-modal features and text
descriptions of actions and objects. We train our model in two stages, where
the model first learns to align video clips with descriptions of future
actions, and is subsequently fine-tuned to predict future actions. Compared to
existing methods, M-CAT has the advantage of learning additional context from
two types of text inputs: rich descriptions of future actions during
pre-training, and, text descriptions for detected objects and actions during
modality feature fusion. Through extensive experimental evaluation, we
demonstrate that our model outperforms previous methods on the EpicKitchens
datasets, and show that using simple text descriptions of actions and objects
aid in more effective action anticipation. In addition, we examine the impact
of object and action information obtained via text, and perform extensive
ablations.
