  The structured time series (STS) classification problem requires the modeling
of interweaved spatiotemporal dependency. most previous STS classification
methods model the spatial and temporal dependencies independently. Due to the
complexity of the STS data, we argue that a desirable STS classification method
should be a holistic framework that can be made as adaptive and flexible as
possible. This motivates us to design a deep neural network with such merits.
Inspired by the dual-stream hypothesis in neural science, we propose a novel
dual-stream framework for modeling the interweaved spatiotemporal dependency,
and develop a convolutional neural network within this framework that aims to
achieve high adaptability and flexibility in STS configurations from various
diagonals, i.e., sequential order, dependency range and features. The proposed
architecture is highly modularized and scalable, making it easy to be adapted
to specific tasks. The effectiveness of our model is demonstrated through
experiments on synthetic data as well as benchmark datasets for skeleton based
activity recognition.
