  Compact convolutional neural networks gain efficiency mainly through
depthwise convolutions, expanded channels and complex topologies, which
contrarily aggravate the training process. Besides, 3x3 kernels dominate the
spatial representation in these models, whereas even-sized kernels (2x2, 4x4)
are rarely adopted. In this work, we quantify the shift problem occurs in
even-sized kernel convolutions by an information erosion hypothesis, and
eliminate it by proposing symmetric padding on four sides of the feature maps
(C2sp, C4sp). Symmetric padding releases the generalization capabilities of
even-sized kernels at little computational cost, making them outperform 3x3
kernels in image classification and generation tasks. Moreover, C2sp obtains
comparable accuracy to emerging compact models with much less memory and time
consumption during training. Symmetric padding coupled with even-sized
convolutions can be neatly implemented into existing frameworks, providing
effective elements for architecture designs, especially on online and continual
learning occasions where training efforts are emphasized.
