  Is it possible to guess human action from dialogue alone? In this work we
investigate the link between spoken words and actions in movies. We note that
movie screenplays describe actions, as well as contain the speech of characters
and hence can be used to learn this correlation with no additional supervision.
We train a BERT-based Speech2Action classifier on over a thousand movie
screenplays, to predict action labels from transcribed speech segments. We then
apply this model to the speech segments of a large unlabelled movie corpus
(188M speech segments from 288K movies). Using the predictions of this model,
we obtain weak action labels for over 800K video clips. By training on these
video clips, we demonstrate superior action recognition performance on standard
action recognition benchmarks, without using a single manually labelled action
example.
