  Designing an effective channel attention mechanism implores one to find a
lossy-compression method allowing for optimal feature representation. Despite
recent progress in the area, it remains an open problem. FcaNet, the current
state-of-the-art channel attention mechanism, attempted to find such an
information-rich compression using Discrete Cosine Transforms (DCTs). One
drawback of FcaNet is that there is no natural choice of the DCT frequencies.
To circumvent this issue, FcaNet experimented on ImageNet to find optimal
frequencies. We hypothesize that the choice of frequency plays only a
supporting role and the primary driving force for the effectiveness of their
attention filters is the orthogonality of the DCT kernels. To test this
hypothesis, we construct an attention mechanism using randomly initialized
orthogonal filters. Integrating this mechanism into ResNet, we create OrthoNet.
We compare OrthoNet to FcaNet (and other attention mechanisms) on Birds,
MS-COCO, and Places356 and show superior performance. On the ImageNet dataset,
our method competes with or surpasses the current state-of-the-art. Our results
imply that an optimal choice of filter is elusive and generalization can be
achieved with a sufficiently large number of orthogonal filters. We further
investigate other general principles for implementing channel attention, such
as its position in the network and channel groupings. Our code is publicly
available at https://github.com/hady1011/OrthoNets/
