  Convolutional neural network (CNN) has achieved impressive success in
computer vision during the past few decades. The image convolution operation
helps CNNs to get good performance on image-related tasks. However, it also has
high computation complexity and hard to be parallelized. This paper proposes a
novel Element-wise Multiplication Layer (EML) to replace convolution layers,
which can be trained in the frequency domain. Theoretical analyses show that
EMLs lower the computation complexity and easier to be parallelized. Moreover,
we introduce a Weight Fixation mechanism to alleviate the problem of
over-fitting, and analyze the working behavior of Batch Normalization and
Dropout in the frequency domain. To get the balance between the computation
complexity and memory usage, we propose a new network structure, namely
Time-Frequency Domain Mixture Network (TFDMNet), which combines the advantages
of both convolution layers and EMLs. Experimental results imply that TFDMNet
achieves good performance on MNIST, CIFAR-10 and ImageNet databases with less
number of operations comparing with corresponding CNNs.
