  Many real-world video analysis applications require the ability to identify
domain-specific events in video, such as interviews and commercials in TV news
broadcasts, or action sequences in film. Unfortunately, pre-trained models to
detect all the events of interest in video may not exist, and training new
models from scratch can be costly and labor-intensive. In this paper, we
explore the utility of specifying new events in video in a more traditional
manner: by writing queries that compose outputs of existing, pre-trained
models. To write these queries, we have developed Rekall, a library that
exposes a data model and programming model for compositional video event
specification. Rekall represents video annotations from different sources
(object detectors, transcripts, etc.) as spatiotemporal labels associated with
continuous volumes of spacetime in a video, and provides operators for
composing labels into queries that model new video events. We demonstrate the
use of Rekall in analyzing video from cable TV news broadcasts, films,
static-camera vehicular video streams, and commercial autonomous vehicle logs.
In these efforts, domain experts were able to quickly (in a few hours to a day)
author queries that enabled the accurate detection of new events (on par with,
and in some cases much more accurate than, learned approaches) and to rapidly
retrieve video clips for human-in-the-loop tasks such as video content curation
and training data curation. Finally, in a user study, novice users of Rekall
were able to author queries to retrieve new events in video given just one hour
of query development time.
