  Modern transformer-based Large Language Models (LLMs) are constructed with a
series of decoder blocks. Each block comprises three key components: (1) QKV
generation, (2) multi-head attention, and (3) feed-forward networks. In batched
processing, QKV generation and feed-forward networks involve compute-intensive
matrix-matrix multiplications (GEMM), while multi-head attention requires
bandwidth-heavy matrix-vector multiplications (GEMV). Machine learning
accelerators like TPUs or NPUs are proficient in handling GEMM but are less
efficient for GEMV computations. Conversely, Processing-in-Memory (PIM)
technology is tailored for efficient GEMV computation, while it lacks the
computational power to handle GEMM effectively. Inspired by this insight, we
propose NeuPIMs, a heterogeneous acceleration system that jointly exploits a
conventional GEMM-focused NPU and GEMV-optimized PIM devices. The main
challenge in efficiently integrating NPU and PIM lies in enabling concurrent
operations on both platforms, each addressing a specific kernel type. First,
existing PIMs typically operate in a "blocked" mode, allowing only either NPU
or PIM to be active at any given time. Second, the inherent dependencies
between GEMM and GEMV in LLMs restrict their parallel processing. To tackle
these challenges, NeuPIMs is equipped with dual row buffers in each bank,
facilitating the simultaneous management of memory read/write operations and
PIM commands. Further, NeuPIMs employs a runtime sub-batch interleaving
technique to maximize concurrent execution, leveraging batch parallelism to
allow two independent sub-batches to be pipelined within a single NeuPIMs
device. Our evaluation demonstrates that compared to GPU-only, NPU-only, and a
na\"ive NPU+PIM integrated acceleration approaches, NeuPIMs achieves 3$\times$,
2.4$\times$ and 1.6$\times$ throughput improvement, respectively.
