  Inspired by the ConvNets with structured hidden representations, we propose a
Tensor-based Neural Network, TCNN. Different from ConvNets, TCNNs are composed
of structured neurons rather than scalar neurons, and the basic operation is
neuron tensor transformation. Unlike other structured ConvNets, where the
part-whole relationships are modeled explicitly, the relationships are learned
implicitly in TCNNs. Also, the structured neurons in TCNNs are high-rank
tensors rather than vectors or matrices. We compare TCNNs with current popular
ConvNets, including ResNets, MobileNets, EfficientNets, RegNets, etc., on
CIFAR10, CIFAR100, and Tiny ImageNet. The experiment shows that TCNNs have
higher efficiency in terms of parameters. TCNNs also show higher robustness
against white-box adversarial attacks on MNIST compared to ConvNets.
