  Unsupervised/self-supervised time series representation learning is a
challenging problem because of its complex dynamics and sparse annotations.
Existing works mainly adopt the framework of contrastive learning with the
time-based augmentation techniques to sample positives and negatives for
contrastive training. Nevertheless, they mostly use segment-level augmentation
derived from time slicing, which may bring about sampling bias and incorrect
optimization with false negatives due to the loss of global context. Besides,
they all pay no attention to incorporate the spectral information in feature
representation. In this paper, we propose a unified framework, namely Bilinear
Temporal-Spectral Fusion (BTSF). Specifically, we firstly utilize the
instance-level augmentation with a simple dropout on the entire time series for
maximally capturing long-term dependencies. We devise a novel iterative
bilinear temporal-spectral fusion to explicitly encode the affinities of
abundant time-frequency pairs, and iteratively refines representations in a
fusion-and-squeeze manner with Spectrum-to-Time (S2T) and Time-to-Spectrum
(T2S) Aggregation modules. We firstly conducts downstream evaluations on three
major tasks for time series including classification, forecasting and anomaly
detection. Experimental results shows that our BTSF consistently significantly
outperforms the state-of-the-art methods.
