  Transformer-based language models utilize the attention mechanism for
substantial performance improvements in almost all natural language processing
(NLP) tasks. Similar attention structures are also extensively studied in
several other areas. Although the attention mechanism enhances the model
performances significantly, its quadratic complexity prevents efficient
processing of long sequences. Recent works focused on eliminating the
disadvantages of computational inefficiency and showed that transformer-based
models can still reach competitive results without the attention layer. A
pioneering study proposed the FNet, which replaces the attention layer with the
Fourier Transform (FT) in the transformer encoder architecture. FNet achieves
competitive performances concerning the original transformer encoder model
while accelerating training process by removing the computational burden of the
attention mechanism. However, the FNet model ignores essential properties of
the FT from the classical signal processing that can be leveraged to increase
model efficiency further. We propose different methods to deploy FT efficiently
in transformer encoder models. Our proposed architectures have smaller number
of model parameters, shorter training times, less memory usage, and some
additional performance improvements. We demonstrate these improvements through
extensive experiments on common benchmarks.
