  Ultra-fast \& low-power superconductor single-flux-quantum (SFQ)-based CNN
systolic accelerators are built to enhance the CNN inference throughput.
However, shift-register (SHIFT)-based scratchpad memory (SPM) arrays prevent a
SFQ CNN accelerator from exceeding 40\% of its peak throughput, due to the lack
of random access capability. This paper first documents our study of a variety
of cryogenic memory technologies, including Vortex Transition Memory (VTM),
Josephson-CMOS SRAM, MRAM, and Superconducting Nanowire Memory, during which we
found that none of the aforementioned technologies made a SFQ CNN accelerator
achieve high throughput, small area, and low power simultaneously. Second, we
present a heterogeneous SPM architecture, SMART, composed of SHIFT arrays and a
random access array to improve the inference throughput of a SFQ CNN systolic
accelerator. Third, we propose a fast, low-power and dense pipelined random
access CMOS-SFQ array by building SFQ passive-transmission-line-based H-Trees
that connect CMOS sub-banks. Finally, we create an ILP-based compiler to deploy
CNN models on SMART. Experimental results show that, with the same chip area
overhead, compared to the latest SHIFT-based SFQ CNN accelerator, SMART
improves the inference throughput by $3.9\times$ ($2.2\times$), and reduces the
inference energy by $86\%$ ($71\%$) when inferring a single image (a batch of
images).
