  Despite the impressive achievements of pre-trained models in the fields of
natural language processing (NLP) and computer vision (CV), progress in the
domain of time series analysis has been limited. In contrast to NLP and CV,
where a single model can handle various tasks, time series analysis still
relies heavily on task-specific methods for activities such as classification,
anomaly detection, forecasting, and few-shot learning. The primary obstacle to
developing a pre-trained model for time series analysis is the scarcity of
sufficient training data. In our research, we overcome this obstacle by
utilizing pre-trained models from language or CV, which have been trained on
billions of data points, and apply them to time series analysis. We assess the
effectiveness of the pre-trained transformer model in two ways. Initially, we
maintain the original structure of the self-attention and feedforward layers in
the residual blocks of the pre-trained language or image model, using the
Frozen Pre-trained Transformer (FPT) for time series analysis with the addition
of projection matrices for input and output. Additionally, we introduce four
unique adapters, designed specifically for downstream tasks based on the
pre-trained model, including forecasting and anomaly detection. These adapters
are further enhanced with efficient parameter tuning, resulting in superior
performance compared to all state-of-the-art methods.Our comprehensive
experimental studies reveal that (a) the simple FPT achieves top-tier
performance across various time series analysis tasks; and (b) fine-tuning the
FPT with the custom-designed adapters can further elevate its performance,
outshining specialized task-specific models.
