  In this paper, we propose a novel layer based on fast Walsh-Hadamard
transform (WHT) and smooth-thresholding to replace $1\times 1$ convolution
layers in deep neural networks. In the WHT domain, we denoise the transform
domain coefficients using the new smooth-thresholding non-linearity, a smoothed
version of the well-known soft-thresholding operator. We also introduce a
family of multiplication-free operators from the basic 2$\times$2 Hadamard
transform to implement $3\times 3$ depthwise separable convolution layers.
Using these two types of layers, we replace the bottleneck layers in
MobileNet-V2 to reduce the network's number of parameters with a slight loss in
accuracy. For example, by replacing the final third bottleneck layers, we
reduce the number of parameters from 2.270M to 540K. This reduces the accuracy
from 95.21\% to 92.98\% on the CIFAR-10 dataset. Our approach significantly
improves the speed of data processing. The fast Walsh-Hadamard transform has a
computational complexity of $O(m\log_2 m)$. As a result, it is computationally
more efficient than the $1\times1$ convolution layer. The fast Walsh-Hadamard
layer processes a tensor in $\mathbb{R}^{10\times32\times32\times1024}$ about 2
times faster than $1\times1$ convolution layer on NVIDIA Jetson Nano computer
board.
