  Existing approaches for learning representations of time-series keep the
temporal arrangement of the time-steps intact with the presumption that the
original order is the most optimal for learning. However, non-adjacent sections
of real-world time-series may have strong dependencies. Accordingly we raise
the question: Is there an alternative arrangement for time-series which could
enable more effective representation learning? To address this, we propose a
simple plug-and-play mechanism called Segment, Shuffle, and Stitch (S3)
designed to improve time-series representation learning of existing models. S3
works by creating non-overlapping segments from the original sequence and
shuffling them in a learned manner that is the most optimal for the task at
hand. It then re-attaches the shuffled segments back together and performs a
learned weighted sum with the original input to capture both the newly shuffled
sequence along with the original sequence. S3 is modular and can be stacked to
create various degrees of granularity, and can be added to many forms of neural
architectures including CNNs or Transformers with negligible computation
overhead. Through extensive experiments on several datasets and
state-of-the-art baselines, we show that incorporating S3 results in
significant improvements for the tasks of time-series classification and
forecasting, improving performance on certain datasets by up to 68\%. We also
show that S3 makes the learning more stable with a smoother training loss curve
and loss landscape compared to the original baseline. The code is available at
https://github.com/shivam-grover/S3-TimeSeries .
