  We propose Attentive Regularization (AR), a method to constrain the
activation maps of kernels in Convolutional Neural Networks (CNNs) to specific
regions of interest (ROIs). Each kernel learns a location of specialization
along with its weights through standard backpropagation. A differentiable
attention mechanism requiring no additional supervision is used to optimize the
ROIs. Traditional CNNs of different types and structures can be modified with
this idea into equivalent Targeted Kernel Networks (TKNs), while keeping the
network size nearly identical. By restricting kernel ROIs, we reduce the number
of sliding convolutional operations performed throughout the network in its
forward pass, speeding up both training and inference. We evaluate our proposed
architecture on both synthetic and natural tasks across multiple domains. TKNs
obtain significant improvements over baselines, requiring less computation
(around an order of magnitude) while achieving superior performance.
