  Video anomaly detection deals with the recognition of abnormal events in
videos. Apart from the visual signal, video anomaly detection has also been
addressed with the use of skeleton sequences. We propose a holistic
representation of skeleton trajectories to learn expected motions across
segments at different times. Our approach uses multitask learning to
reconstruct any continuous unobserved temporal segment of the trajectory
allowing the extrapolation of past or future segments and the interpolation of
in-between segments. We use an end-to-end attention-based encoder-decoder. We
encode temporally occluded trajectories, jointly learn latent representations
of the occluded segments, and reconstruct trajectories based on expected
motions across different temporal segments. Extensive experiments on three
trajectory-based video anomaly detection datasets show the advantages and
effectiveness of our approach with state-of-the-art results on anomaly
detection in skeleton trajectories.
