  We consider Convolutional Neural Networks (CNNs) with 2D structured features
that are symmetric in the spatial dimensions. Such networks arise in modeling
pairwise relationships for a sequential recommendation problem, as well as
secondary structure inference problems of RNA and protein sequences. We develop
a CNN architecture that generates and preserves the symmetry structure in the
network's convolutional layers. We present parameterizations for the
convolutional kernels that produce update rules to maintain symmetry throughout
the training. We apply this architecture to the sequential recommendation
problem, the RNA secondary structure inference problem, and the protein contact
map prediction problem, showing that the symmetric structured networks produce
improved results using fewer numbers of machine parameters.
