  Convolutional layers are a major driving force behind the successes of deep
learning. Pointwise convolution (PWC) is a 1x1 convolutional filter that is
primarily used for parameter reduction. However, the PWC ignores the spatial
information around the points it is processing. This design is by choice, in
order to reduce the overall parameters and computations. However, we
hypothesize that this shortcoming of PWC has a significant impact on the
network performance. We propose an alternative design for pointwise
convolution, which uses spatial information from the input efficiently. Our
design significantly improves the performance of the networks without
substantially increasing the number of parameters and computations. We
experimentally show that our design results in significant improvement in the
performance of the network for classification as well as detection.
