  Time series analysis is widely used in extensive areas. Recently, to reduce
labeling expenses and benefit various tasks, self-supervised pre-training has
attracted immense interest. One mainstream paradigm is masked modeling, which
successfully pre-trains deep models by learning to reconstruct the masked
content based on the unmasked part. However, since the semantic information of
time series is mainly contained in temporal variations, the standard way of
randomly masking a portion of time points will seriously ruin vital temporal
variations of time series, making the reconstruction task too difficult to
guide representation learning. We thus present SimMTM, a Simple pre-training
framework for Masked Time-series Modeling. By relating masked modeling to
manifold learning, SimMTM proposes to recover masked time points by the
weighted aggregation of multiple neighbors outside the manifold, which eases
the reconstruction task by assembling ruined but complementary temporal
variations from multiple masked series. SimMTM further learns to uncover the
local structure of the manifold, which is helpful for masked modeling.
Experimentally, SimMTM achieves state-of-the-art fine-tuning performance
compared to the most advanced time series pre-training methods in two canonical
time series analysis tasks: forecasting and classification, covering both in-
and cross-domain settings.
