  From fully connected neural networks to convolutional neural networks, the
learned parameters within a neural network have been primarily relegated to the
linear parameters (e.g., convolutional filters). The non-linear functions
(e.g., activation functions) have largely remained, with few exceptions in
recent years, parameter-less, static throughout training, and seen limited
variation in design. Largely ignored by the deep learning community, radial
basis function (RBF) networks provide an interesting mechanism for learning
more complex non-linear activation functions in addition to the linear
parameters in a network. However, the interest in RBF networks has waned over
time due to the difficulty of integrating RBFs into more complex deep neural
network architectures in a tractable and stable manner. In this work, we
present a novel approach that enables end-to-end learning of deep RBF networks
with fully learnable activation basis functions in an automatic and tractable
manner. We demonstrate that our approach for enabling the use of learnable
activation basis functions in deep neural networks, which we will refer to as
DeepLABNet, is an effective tool for automated activation function learning
within complex network architectures.
