  Combining the Color and Event cameras (also called Dynamic Vision Sensors,
DVS) for robust object tracking is a newly emerging research topic in recent
years. Existing color-event tracking framework usually contains multiple
scattered modules which may lead to low efficiency and high computational
complexity, including feature extraction, fusion, matching, interactive
learning, etc. In this paper, we propose a single-stage backbone network for
Color-Event Unified Tracking (CEUTrack), which achieves the above functions
simultaneously. Given the event points and RGB frames, we first transform the
points into voxels and crop the template and search regions for both
modalities, respectively. Then, these regions are projected into tokens and
parallelly fed into the unified Transformer backbone network. The output
features will be fed into a tracking head for target object localization. Our
proposed CEUTrack is simple, effective, and efficient, which achieves over 75
FPS and new SOTA performance. To better validate the effectiveness of our model
and address the data deficiency of this task, we also propose a generic and
large-scale benchmark dataset for color-event tracking, termed COESOT, which
contains 90 categories and 1354 video sequences. Additionally, a new evaluation
metric named BOC is proposed in our evaluation toolkit to evaluate the
prominence with respect to the baseline methods. We hope the newly proposed
method, dataset, and evaluation metric provide a better platform for
color-event-based tracking. The dataset, toolkit, and source code will be
released on: \url{https://github.com/Event-AHU/COESOT}.
