  Convolutional networks are not aware of an object's geometric variations,
which leads to inefficient utilization of model and data capacity. To overcome
this issue, recent works on deformation modeling seek to spatially reconfigure
the data towards a common arrangement such that semantic recognition suffers
less from deformation. This is typically done by augmenting static operators
with learned free-form sampling grids in the image space, dynamically tuned to
the data and task for adapting the receptive field. Yet adapting the receptive
field does not quite reach the actual goal -- what really matters to the
network is the "effective" receptive field (ERF), which reflects how much each
pixel contributes. It is thus natural to design other approaches to adapt the
ERF directly during runtime.
  In this work, we instantiate one possible solution as Deformable Kernels
(DKs), a family of novel and generic convolutional operators for handling
object deformations by directly adapting the ERF while leaving the receptive
field untouched. At the heart of our method is the ability to resample the
original kernel space towards recovering the deformation of objects. This
approach is justified with theoretical insights that the ERF is strictly
determined by data sampling locations and kernel values. We implement DKs as
generic drop-in replacements of rigid kernels and conduct a series of empirical
studies whose results conform with our theories. Over several tasks and
standard base models, our approach compares favorably against prior works that
adapt during runtime. In addition, further experiments suggest a working
mechanism orthogonal and complementary to previous works.
