  We present a novel and compact architecture for deep Convolutional Neural
Networks (CNNs) in this paper, termed $3$D-FilterMap Convolutional Neural
Networks ($3$D-FM-CNNs). The convolution layer of $3$D-FM-CNN learns a compact
representation of the filters, named $3$D-FilterMap, instead of a set of
independent filters in the conventional convolution layer. The filters are
extracted from the $3$D-FilterMap as overlapping $3$D submatrics with weight
sharing among nearby filters, and these filters are convolved with the input to
generate the output of the convolution layer for $3$D-FM-CNN. Due to the weight
sharing scheme, the parameter size of the $3$D-FilterMap is much smaller than
that of the filters to be learned in the conventional convolution layer when
$3$D-FilterMap generates the same number of filters. Our work is fundamentally
different from the network compression literature that reduces the size of a
learned large network in the sense that a small network is directly learned
from scratch. Experimental results demonstrate that $3$D-FM-CNN enjoys a small
parameter space by learning compact $3$D-FilterMaps, while achieving
performance compared to that of the baseline CNNs which learn the same number
of filters as that generated by the corresponding $3$D-FilterMap.
