  Learning time-series representations for discriminative tasks has been a
long-standing challenge. Current pre-training methods are limited in either
unidirectional next-token prediction or randomly masked token prediction. We
propose a novel architecture called Bidirectional Timely Generative Pre-trained
Transformer (BiTimelyGPT), which pre-trains on time-series data by both
next-token and previous-token predictions in alternating transformer layers.
This pre-training task preserves original distribution and data shapes of the
time-series. Additionally, the full-rank forward and backward attention
matrices exhibit more expressive representation capabilities. Using biosignal
data, BiTimelyGPT demonstrates superior performance in predicting neurological
functionality, disease diagnosis, and physiological signs. By visualizing the
attention heatmap, we observe that the pre-trained BiTimelyGPT can identify
discriminative segments from time-series sequences, even more so after
fine-tuning on the task.
