  The non-commutative harmonic oscillator (NCHO) is a matrix valued
differential operator originally introduced as a generalization of the quantum
harmonic oscillator having a weaker $\mathfrak{sl}_2(\mathbb{R})$-symmetry. The
spectrum of the NCHO has remarkable properties, including the presence of
number theoretical structures such as modular forms, elliptic curves and
Eichler cohomology observed in the special values of the associated spectral
zeta function. In addition, the Heun ODE picture of the eigenvalue problem of
the NCHO reveals a connection with the quantum Rabi model (QRM), a fundamental
interaction model from quantum optics. In this paper we introduce an
$\eta$-shifted NCHO ($\eta$-NCHO) that has an analogous relation with the
asymmetric quantum Rabi model (AQRM) and describe its basic properties. Even
though the shift factor does not break the parity symmetry of the NCHO, a
certain type of degeneracies appears for $\eta \in \frac12 \mathbb{Z}$, as if
mirroring the situation of the AQRM. We give furthermore a detailed description
of the confluence process, that we call iso-parallel confluence process due to
the fact that it requires a parallel transformation of two parameters
describing the spectrum of $\eta$-NCHO and representations of
$\mathfrak{sl}_2(\mathbb{R})$. We relate the eigenvalues of the two models
under the iso-parallel confluence process, including how the quasi-exact
eigenfunctions of the $\eta$-NCHO correspond to Juddian solutions of the AQRM.
From the point of view of this confluence process, a family of $\eta$-NCHO
corresponds to a single AQRM, thus we may regard the $\eta$-NCHO as a covering
of the AQRM. We expect the study of the $\eta$-NCHO and the AQRM from this
point of view to be helpful for the clarification of several questions on the
AQRM, including the hidden symmetry and the number of Juddian solutions.
