  In this paper, we introduce MINTIME, a video deepfake detection approach that
captures spatial and temporal anomalies and handles instances of multiple
people in the same video and variations in face sizes. Previous approaches
disregard such information either by using simple a-posteriori aggregation
schemes, i.e., average or max operation, or using only one identity for the
inference, i.e., the largest one. On the contrary, the proposed approach builds
on a Spatio-Temporal TimeSformer combined with a Convolutional Neural Network
backbone to capture spatio-temporal anomalies from the face sequences of
multiple identities depicted in a video. This is achieved through an
Identity-aware Attention mechanism that attends to each face sequence
independently based on a masking operation and facilitates video-level
aggregation. In addition, two novel embeddings are employed: (i) the Temporal
Coherent Positional Embedding that encodes each face sequence's temporal
information and (ii) the Size Embedding that encodes the size of the faces as a
ratio to the video frame size. These extensions allow our system to adapt
particularly well in the wild by learning how to aggregate information of
multiple identities, which is usually disregarded by other methods in the
literature. It achieves state-of-the-art results on the ForgeryNet dataset with
an improvement of up to 14% AUC in videos containing multiple people and
demonstrates ample generalization capabilities in cross-forgery and
cross-dataset settings. The code is publicly available at
https://github.com/davide-coccomini/MINTIME-Multi-Identity-size-iNvariant-TIMEsformer-for-Video-Deepfake-Detection.
