  Recent work in neural networks for image classification has seen a strong
tendency towards increasing the spatial context. Whether achieved through large
convolution kernels or self-attention, models scale poorly with the increased
spatial context, such that the improved model accuracy often comes at
significant costs. In this paper, we propose a module for studying the
effective filter size of convolutional neural networks. To facilitate such a
study, several challenges need to be addressed: 1) we need an effective means
to train models with large filters (potentially as large as the input data)
without increasing the number of learnable parameters 2) the employed
convolution operation should be a plug-and-play module that can replace
conventional convolutions in a CNN and allow for an efficient implementation in
current frameworks 3) the study of filter sizes has to be decoupled from other
aspects such as the network width or the number of learnable parameters 4) the
cost of the convolution operation itself has to remain manageable i.e. we
cannot naively increase the size of the convolution kernel. To address these
challenges, we propose to learn the frequency representations of filter weights
as neural implicit functions, such that the better scalability of the
convolution in the frequency domain can be leveraged. Additionally, due to the
implementation of the proposed neural implicit function, even large and
expressive spatial filters can be parameterized by only a few learnable
weights. Our analysis shows that, although the proposed networks could learn
very large convolution kernels, the learned filters are well localized and
relatively small in practice when transformed from the frequency to the spatial
domain. We anticipate that our analysis of individually optimized filter sizes
will allow for more efficient, yet effective, models in the future.
https://github.com/GeJulia/NIFF.
