  Analog in-memory computing (AIMC) cores offers significant performance and
energy benefits for neural network inference with respect to digital logic
(e.g., CPUs). AIMCs accelerate matrix-vector multiplications, which dominate
these applications' run-time. However, AIMC-centric platforms lack the
flexibility of general-purpose systems, as they often have hard-coded data
flows and can only support a limited set of processing functions. With the goal
of bridging this gap in flexibility, we present a novel system architecture
that tightly integrates analog in-memory computing accelerators into multi-core
CPUs in general-purpose systems. We developed a powerful gem5-based full
system-level simulation framework into the gem5-X simulator, ALPINE, which
enables an in-depth characterization of the proposed architecture. ALPINE
allows the simulation of the entire computer architecture stack from major
hardware components to their interactions with the Linux OS. Within ALPINE, we
have defined a custom ISA extension and a software library to facilitate the
deployment of inference models. We showcase and analyze a variety of mappings
of different neural network types, and demonstrate up to 20.5x/20.8x
performance/energy gains with respect to a SIMD-enabled ARM CPU implementation
for convolutional neural networks, multi-layer perceptrons, and recurrent
neural networks.
