  Recent research on time-series self-supervised models shows great promise in
learning semantic representations. However, it has been limited to small-scale
datasets, e.g., thousands of temporal sequences. In this work, we make key
technical contributions that are tailored to the numerical properties of
time-series data and allow the model to scale to large datasets, e.g., millions
of temporal sequences. We adopt the Transformer architecture by first
partitioning the input into non-overlapping windows. Each window is then
characterized by its normalized shape and two scalar values denoting the mean
and standard deviation within each window. To embed scalar values that may
possess arbitrary numerical amplitudes in a high-dimensional space, we propose
a numerically multi-scaled embedding module enumerating all possible numerical
scales for the scalars. The model undergoes pretraining with a simple
contrastive objective on a large-scale dataset over a million sequences
collected by merging existing public data. We study its transfer performance on
a number of univariate and multivariate classification tasks, few shot
learning, unsupervised clustering and anomaly detection benchmarks. Our method
exhibits remarkable improvement against previous pretraining approaches and
establishes the new state of the art, even compared with domain-specific
non-learning-based methods. Code is available at:
\url{https://github.com/chenguolin/NuTime}.
