  As tools for content editing mature, and artificial intelligence (AI) based
algorithms for synthesizing media grow, the presence of manipulated content
across online media is increasing. This phenomenon causes the spread of
misinformation, creating a greater need to distinguish between ``real'' and
``manipulated'' content. To this end, we present VideoSham, a dataset
consisting of 826 videos (413 real and 413 manipulated). Many of the existing
deepfake datasets focus exclusively on two types of facial manipulations --
swapping with a different subject's face or altering the existing face.
VideoSham, on the other hand, contains more diverse, context-rich, and
human-centric, high-resolution videos manipulated using a combination of 6
different spatial and temporal attacks. Our analysis shows that
state-of-the-art manipulation detection algorithms only work for a few specific
attacks and do not scale well on VideoSham. We performed a user study on Amazon
Mechanical Turk with 1200 participants to understand if they can differentiate
between the real and manipulated videos in VideoSham. Finally, we dig deeper
into the strengths and weaknesses of performances by humans and SOTA-algorithms
to identify gaps that need to be filled with better AI algorithms. We present
the dataset at https://github.com/adobe-research/VideoSham-dataset.
