  Small Video Object Detection (SVOD) is a crucial subfield in modern computer
vision, essential for early object discovery and detection. However, existing
SVOD datasets are scarce and suffer from issues such as insufficiently small
objects, limited object categories, and lack of scene diversity, leading to
unitary application scenarios for corresponding methods. To address this gap,
we develop the XS-VID dataset, which comprises aerial data from various periods
and scenes, and annotates eight major object categories. To further evaluate
existing methods for detecting extremely small objects, XS-VID extensively
collects three types of objects with smaller pixel areas: extremely small
(\textit{es}, $0\sim12^2$), relatively small (\textit{rs}, $12^2\sim20^2$), and
generally small (\textit{gs}, $20^2\sim32^2$). XS-VID offers unprecedented
breadth and depth in covering and quantifying minuscule objects, significantly
enriching the scene and object diversity in the dataset. Extensive validations
on XS-VID and the publicly available VisDrone2019VID dataset show that existing
methods struggle with small object detection and significantly underperform
compared to general object detectors. Leveraging the strengths of previous
methods and addressing their weaknesses, we propose YOLOFT, which enhances
local feature associations and integrates temporal motion features,
significantly improving the accuracy and stability of SVOD. Our datasets and
benchmarks are available at \url{https://gjhhust.github.io/XS-VID/}.
