  Distinguishing if an action is performed as intended or if an intended action
fails is an important skill that not only humans have, but that is also
important for intelligent systems that operate in human environments.
Recognizing if an action is unintentional or anticipating if an action will
fail, however, is not straightforward due to lack of annotated data. While
videos of unintentional or failed actions can be found in the Internet in
abundance, high annotation costs are a major bottleneck for learning networks
for these tasks. In this work, we thus study the problem of self-supervised
representation learning for unintentional action prediction. While previous
works learn the representation based on a local temporal neighborhood, we show
that the global context of a video is needed to learn a good representation for
the three downstream tasks: unintentional action classification, localization
and anticipation. In the supplementary material, we show that the learned
representation can be used for detecting anomalies in videos as well.
