  Convolutional neural networks (CNNs) are inherently equivariant to
translation. Efforts to embed other forms of equivariance have concentrated
solely on rotation. We expand the notion of equivariance in CNNs through the
Polar Transformer Network (PTN). PTN combines ideas from the Spatial
Transformer Network (STN) and canonical coordinate representations. The result
is a network invariant to translation and equivariant to both rotation and
scale. PTN is trained end-to-end and composed of three distinct stages: a polar
origin predictor, the newly introduced polar transformer module and a
classifier. PTN achieves state-of-the-art on rotated MNIST and the newly
introduced SIM2MNIST dataset, an MNIST variation obtained by adding clutter and
perturbing digits with translation, rotation and scaling. The ideas of PTN are
extensible to 3D which we demonstrate through the Cylindrical Transformer
Network.
