  Background subtraction (BGS) aims to extract all moving objects in the video
frames to obtain binary foreground segmentation masks. Deep learning has been
widely used in this field. Compared with supervised-based BGS methods,
unsupervised methods have better generalization. However, previous unsupervised
deep learning BGS algorithms perform poorly in sophisticated scenarios such as
shadows or night lights, and they cannot detect objects outside the pre-defined
categories. In this work, we propose an unsupervised BGS algorithm based on
zero-shot object detection called Zero-shot Background Subtraction (ZBS). The
proposed method fully utilizes the advantages of zero-shot object detection to
build the open-vocabulary instance-level background model. Based on it, the
foreground can be effectively extracted by comparing the detection results of
new frames with the background model. ZBS performs well for sophisticated
scenarios, and it has rich and extensible categories. Furthermore, our method
can easily generalize to other tasks, such as abandoned object detection in
unseen environments. We experimentally show that ZBS surpasses state-of-the-art
unsupervised BGS methods by 4.70% F-Measure on the CDnet 2014 dataset. The code
is released at https://github.com/CASIA-IVA-Lab/ZBS.
