  (This paper was written in November 2011 and never published. It is posted on
arXiv.org in its original form in June 2016). Many recent object recognition
systems have proposed using a two phase training procedure to learn sparse
convolutional feature hierarchies: unsupervised pre-training followed by
supervised fine-tuning. Recent results suggest that these methods provide
little improvement over purely supervised systems when the appropriate
nonlinearities are included. This paper presents an empirical exploration of
the space of learning procedures for sparse convolutional networks to assess
which method produces the best performance. In our study, we introduce an
augmentation of the Predictive Sparse Decomposition method that includes a
discriminative term (DPSD). We also introduce a new single phase supervised
learning procedure that places an L1 penalty on the output state of each layer
of the network. This forces the network to produce sparse codes without the
expensive pre-training phase. Using DPSD with a new, complex predictor that
incorporates lateral inhibition, combined with multi-scale feature pooling, and
supervised refinement, the system achieves a 70.6\% recognition rate on
Caltech-101. With the addition of convolutional training, a 77\% recognition
was obtained on the CIfAR-10 dataset.
