  FPGA-based hardware accelerators for convolutional neural networks (CNNs)
have obtained great attentions due to their higher energy efficiency than GPUs.
However, it is challenging for FPGA-based solutions to achieve a higher
throughput than GPU counterparts. In this paper, we demonstrate that FPGA
acceleration can be a superior solution in terms of both throughput and energy
efficiency when a CNN is trained with binary constraints on weights and
activations. Specifically, we propose an optimized FPGA accelerator
architecture tailored for bitwise convolution and normalization that features
massive spatial parallelism with deep pipelines stages. A key advantage of the
FPGA accelerator is that its performance is insensitive to data batch size,
while the performance of GPU acceleration varies largely depending on the batch
size of the data. Experiment results show that the proposed accelerator
architecture for binary CNNs running on a Virtex-7 FPGA is 8.3x faster and 75x
more energy-efficient than a Titan X GPU for processing online individual
requests in small batch sizes. For processing static data in large batch sizes,
the proposed solution is on a par with a Titan X GPU in terms of throughput
while delivering 9.5x higher energy efficiency.
