  We present an approach to labeling short video clips with English verbs as
event descriptions. A key distinguishing aspect of this work is that it labels
videos with verbs that describe the spatiotemporal interaction between event
participants, humans and objects interacting with each other, abstracting away
all object-class information and fine-grained image characteristics, and
relying solely on the coarse-grained motion of the event participants. We apply
our approach to a large set of 22 distinct verb classes and a corpus of 2,584
videos, yielding two surprising outcomes. First, a classification accuracy of
greater than 70% on a 1-out-of-22 labeling task and greater than 85% on a
variety of 1-out-of-10 subsets of this labeling task is independent of the
choice of which of two different time-series classifiers we employ. Second, we
achieve this level of accuracy using a highly impoverished intermediate
representation consisting solely of the bounding boxes of one or two event
participants as a function of time. This indicates that successful event
recognition depends more on the choice of appropriate features that
characterize the linguistic invariants of the event classes than on the
particular classifier algorithms.
