  Background subtraction has been a driving engine for many computer vision and
video analytics tasks. Although its many variants exist, they all share the
underlying assumption that photometric scene properties are either static or
exhibit temporal stationarity. While this works in some applications, the model
fails when one is interested in discovering {\it changes in scene dynamics}
rather than those in a static background; detection of unusual pedestrian and
motor traffic patterns is but one example. We propose a new model and
computational framework that address this failure by considering stationary
scene dynamics as a ``background'' with which observed scene dynamics are
compared. Central to our approach is the concept of an {\it event}, that we
define as short-term scene dynamics captured over a time window at a specific
spatial location in the camera field of view. We compute events by
time-aggregating motion labels, obtained by background subtraction, as well as
object descriptors (e.g., object size). Subsequently, we characterize events
probabilistically, but use a low-memory, low-complexity surrogates in practical
implementation. Using these surrogates amounts to {\it behavior subtraction}, a
new algorithm with some surprising properties. As demonstrated here, behavior
subtraction is an effective tool in anomaly detection and localization. It is
resilient to spurious background motion, such as one due to camera jitter, and
is content-blind, i.e., it works equally well on humans, cars, animals, and
other objects in both uncluttered and highly-cluttered scenes. Clearly,
treating video as a collection of events rather than colored pixels opens new
possibilities for video analytics.
