  Deep networks have shown great performance in classification tasks. However,
the parameters learned by the classifier networks usually discard stylistic
information of the input, in favour of information strictly relevant to
classification. We introduce a network that has the capacity to do both
classification and reconstruction by adding a "style memory" to the output
layer of the network. We also show how to train such a neural network as a deep
multi-layer autoencoder, jointly minimizing both classification and
reconstruction losses. The generative capacity of our network demonstrates that
the combination of style-memory neurons with the classifier neurons yield good
reconstructions of the inputs when the classification is correct. We further
investigate the nature of the style memory, and how it relates to composing
digits and letters. Finally, we propose that this architecture enables the
bidirectional flow of information used in predictive coding, and that such
bidirectional networks can help mitigate against being fooled by ambiguous or
adversarial input.
