  Structured dense matrices result from boundary integral problems in
electrostatics and geostatistics, and also Schur complements in sparse
preconditioners such as multi-frontal methods. Exploiting the structure of such
matrices can reduce the time for dense direct factorization from $O(N^3)$ to
$O(N)$. The Hierarchically Semi-Separable (HSS) matrix is one such low rank
matrix format that can be factorized using a Cholesky-like algorithm called ULV
factorization. The HSS-ULV algorithm is highly parallel because it removes the
dependency on trailing sub-matrices at each HSS level. However, a key merge
step that links two successive HSS levels remains a challenge for efficient
parallelization. In this paper, we use an asynchronous runtime system PaRSEC
with the HSS-ULV algorithm. We compare our work with STRUMPACK and LORAPO, both
state-of-the-art implementations of dense direct low rank factorization, and
achieve up to 2x better factorization time for matrices arising from a diverse
set of applications on up to 128 nodes of Fugaku for similar or better accuracy
for all the problems that we survey.
