  Even with generational improvements in DRAM technology, memory access latency
still remains the major bottleneck for application accelerators, primarily due
to limitations in memory interface IPs which cannot fully account for
variations in target applications, the algorithms used, and accelerator
architectures. Since developing memory controllers for different applications
is time-consuming, this paper introduces a modular and programmable memory
controller that can be configured for different target applications on
available hardware resources. The proposed memory controller efficiently
supports cache-line accesses along with bulk memory transfers. The user can
configure the controller depending on the available logic resources on the
FPGA, memory access pattern, and external memory specifications. The modular
design supports various memory access optimization techniques including,
request scheduling, internal caching, and direct memory access. These
techniques contribute to reducing the overall latency while maintaining high
sustained bandwidth. We implement the system on a state-of-the-art FPGA and
evaluate its performance using two widely studied domains: graph analytics and
deep learning workloads. We show improved overall memory access time up to 58%
on CNN and GCN workloads compared with commercial memory controller IPs.
