  Deep convolutional networks are vulnerable to image translation or shift,
partly due to common down-sampling layers, e.g., max-pooling and strided
convolution. These operations violate the Nyquist sampling rate and cause
aliasing. The textbook solution is low-pass filtering (blurring) before
down-sampling, which can benefit deep networks as well. Even so, non-linearity
units, such as ReLU, often re-introduce the problem, suggesting that blurring
alone may not suffice. In this work, first, we analyse deep features with
Fourier transform and show that Depth Adaptive Blurring is more effective, as
opposed to monotonic blurring. To this end, we outline how this can replace
existing down-sampling methods. Second, we introduce a novel activation
function -- with a built-in low pass filter, to keep the problem from
reappearing. From experiments, we observe generalisation on other forms of
transformations and corruptions as well, e.g., rotation, scale, and noise. We
evaluate our method under three challenging settings: (1) a variety of image
translations; (2) adversarial attacks -- both $\ell_{p}$ bounded and unbounded;
and (3) data corruptions and perturbations. In each setting, our method
achieves state-of-the-art results and improves clean accuracy on various
benchmark datasets.
