  Attention operators have been applied on both 1-D data like texts and
higher-order data such as images and videos. Use of attention operators on
high-order data requires flattening of the spatial or spatial-temporal
dimensions into a vector, which is assumed to follow a multivariate normal
distribution. This not only incurs excessive requirements on computational
resources, but also fails to preserve structures in data. In this work, we
propose to avoid flattening by assuming the data follow matrix-variate normal
distributions. Based on this new view, we develop Kronecker attention operators
(KAOs) that operate on high-order tensor data directly. More importantly, the
proposed KAOs lead to dramatic reductions in computational resources.
Experimental results show that our methods reduce the amount of required
computational resources by a factor of hundreds, with larger factors for
higher-dimensional and higher-order data. Results also show that networks with
KAOs outperform models without attention, while achieving competitive
performance as those with original attention operators.
