  Pooling is an important component in convolutional neural networks (CNNs) for
aggregating features and reducing computational burden. Compared with other
components such as convolutional layers and fully connected layers which are
completely learned from data, the pooling component is still handcrafted such
as max pooling and average pooling. This paper proposes a learnable pooling
function using recurrent neural networks (RNN) so that the pooling can be fully
adapted to data and other components of the network, leading to an improved
performance. Such a network with learnable pooling function is referred to as a
fully trainable network (FTN). Experimental results have demonstrated that the
proposed RNN-based pooling can well approximate the existing pooling functions
and improve the performance of the network. Especially for small networks, the
proposed FTN can improve the performance by seven percentage points in terms of
error rate on the CIFAR-10 dataset compared with the traditional CNN.
