  Maritime surveillance is vital to mitigate illegal activities such as drug
smuggling, illegal fishing, and human trafficking. Vision-based maritime
surveillance is challenging mainly due to visibility issues at night, which
results in failures in re-identifying vessels and detecting suspicious
activities. In this paper, we introduce a thermal, vision-based approach for
maritime surveillance with object tracking, vessel re-identification, and
suspicious activity detection capabilities. For vessel re-identification, we
propose a novel viewpoint-independent algorithm which compares features of the
sides of the vessel separately (separate side-spaces) leveraging shape
information in the absence of color features. We propose techniques to adapt
tracking and activity detection algorithms for the thermal domain and train
them using a thermal dataset we created. This dataset will be the first
publicly available benchmark dataset for thermal maritime surveillance. Our
system is capable of re-identifying vessels with an 81.8% Top1 score and
identifying suspicious activities with a 72.4\% frame mAP score; a new
benchmark for each task in the thermal domain.
