  Factorization and multiplication of dense matrices and tensors are critical,
yet extremely expensive pieces of the scientific toolbox. Careful use of low
rank approximation can drastically reduce the computation and memory
requirements of these operations. In addition to a lower arithmetic complexity,
such methods can, by their structure, be designed to efficiently exploit modern
hardware architectures. The majority of existing work relies on batched BLAS
libraries to handle the computation of many small dense matrices. We show that
through careful analysis of the cache utilization, register accumulation using
SIMD registers and a redesign of the implementation, one can achieve
significantly higher throughput for these types of batched low-rank matrices
across a large range of block and batch sizes. We test our algorithm on 3 CPUs
using diverse ISAs -- the Fujitsu A64FX using ARM SVE, the Intel Xeon 6148
using AVX-512 and AMD EPYC 7502 using AVX-2, and show that our new batching
methodology is able to obtain more than twice the throughput of vendor
optimized libraries for all CPU architectures and problem sizes.
