  Compared to Multilayer Neural Networks with real weights, Binary Multilayer
Neural Networks (BMNNs) can be implemented more efficiently on dedicated
hardware. BMNNs have been demonstrated to be effective on binary classification
tasks with Expectation BackPropagation (EBP) algorithm on high dimensional text
datasets. In this paper, we investigate the capability of BMNNs using the EBP
algorithm on multiclass image classification tasks. The performances of binary
neural networks with multiple hidden layers and different numbers of hidden
units are examined on MNIST. We also explore the effectiveness of image spatial
filters and the dropout technique in BMNNs. Experimental results on MNIST
dataset show that EBP can obtain 2.12% test error with binary weights and 1.66%
test error with real weights, which is comparable to the results of standard
BackPropagation algorithm on fully connected MNNs.
