  In conventional machine learning applications, each data attribute is assumed
to be orthogonal to others. Namely, every pair of dimension is orthogonal to
each other and thus there is no distinction of in-between relations of
dimensions. However, this is certainly not the case in real world signals which
naturally originate from a spatio-temporal configuration. As a result, the
conventional vectorization process disrupts all of the spatio-temporal
information about the order/place of data whether it be $1$D, $2$D, $3$D, or
$4$D. In this paper, the problem of orthogonality is first investigated through
conventional $k$-means of images, where images are to be processed as vectors.
As a solution, shift-invariant $k$-means is proposed in a novel framework with
the help of sparse representations. A generalization of shift-invariant
$k$-means, convolutional dictionary learning, is then utilized as an
unsupervised feature extraction method for classification. Experiments suggest
that Gabor feature extraction as a simulation of shallow convolutional neural
networks provides a little better performance compared to convolutional
dictionary learning. Many alternatives of convolutional-logic are also
discussed for spatio-temporal information preservation, including a
spatio-temporal hypercomplex encoding scheme.
