  Many deep learning based methods are designed to remove non-uniform
(spatially variant) motion blur caused by object motion and camera shake
without knowing the blur kernel. Some methods directly output the latent sharp
image in one stage, while others utilize a multi-stage strategy (\eg
multi-scale, multi-patch, or multi-temporal) to gradually restore the sharp
image. However, these methods have the following two main issues: 1) The
computational cost of multi-stage is high; 2) The same convolution kernel is
applied in different regions, which is not an ideal choice for non-uniform
blur. Hence, non-uniform motion deblurring is still a challenging and open
problem. In this paper, we propose a new architecture which consists of
multiple Atrous Spatial Pyramid Deformable Convolution (ASPDC) modules to
deblur an image end-to-end with more flexibility. Multiple ASPDC modules
implicitly learn the pixel-specific motion with different dilation rates in the
same layer to handle movements of different magnitude. To improve the training,
we also propose a reblurring network to map the deblurred output back to the
blurred input, which constrains the solution space. Our experimental results
show that the proposed method outperforms state-of-the-art methods on the
benchmark datasets. The code is available at https://github.com/Dong-Huo/ASPDC.
