  The configurable building blocks of current FPGAs -- Logic blocks (LBs),
Digital Signal Processing (DSP) slices, and Block RAMs (BRAMs) -- make them
efficient hardware accelerators for the rapid-changing world of Deep Learning
(DL). Communication between these blocks happens through an interconnect fabric
consisting of switching elements spread throughout the FPGA. In this paper, a
new block, Compute RAM, is proposed. Compute RAMs provide highly-parallel
processing-in-memory (PIM) by combining computation and storage capabilities in
one block. Compute RAMs can be integrated in the FPGA fabric just like the
existing FPGA blocks and provide two modes of operation (storage or compute)
that can be dynamically chosen. They reduce power consumption by reducing data
movement, provide adaptable precision support, and increase the effective
on-chip memory bandwidth. Compute RAMs also help increase the compute density
of FPGAs. In our evaluation of addition, multiplication and dot-product
operations across multiple data precisions (int4, int8 and bfloat16), we
observe an average savings of 80% in energy consumption, and an improvement in
execution time ranging from 20% to 80%. Adding Compute RAMs can benefit non-DL
applications as well, and make FPGAs more efficient, flexible, and performant
accelerators.
