  Deep Convolutional Neural Networks are able to identify complex patterns and
perform tasks with super-human capabilities. However, besides the exceptional
results, they are not completely understood and it is still impractical to
hand-engineer similar solutions. In this work, an image classification
Convolutional Neural Network and its building blocks are described from a
frequency domain perspective. Some network layers have established counterparts
in the frequency domain like the convolutional and pooling layers. We propose
the 2SReLU layer, a novel non-linear activation function that preserves high
frequency components in deep networks. It is demonstrated that in the frequency
domain it is possible to achieve competitive results without using the
computationally costly convolution operation. A source code implementation in
PyTorch is provided at: https://gitlab.com/thomio/2srelu
