  We present a novel end-to-end partially supervised deep learning approach for
video anomaly detection and localization using only normal samples. The insight
that motivates this study is that the normal samples can be associated with at
least one Gaussian component of a Gaussian Mixture Model (GMM), while anomalies
either do not belong to any Gaussian component. The method is based on Gaussian
Mixture Variational Autoencoder, which can learn feature representations of the
normal samples as a Gaussian Mixture Model trained using deep learning. A Fully
Convolutional Network (FCN) that does not contain a fully-connected layer is
employed for the encoder-decoder structure to preserve relative spatial
coordinates between the input image and the output feature map. Based on the
joint probabilities of each of the Gaussian mixture components, we introduce a
sample energy based method to score the anomaly of image test patches. A
two-stream network framework is employed to combine the appearance and motion
anomalies, using RGB frames for the former and dynamic flow images, for the
latter. We test our approach on two popular benchmarks (UCSD Dataset and Avenue
Dataset). The experimental results verify the superiority of our method
compared to the state of the arts.
