  The aim of this paper is a better understanding for the eigenstates of the
asymmetric quantum Rabi model by Lie algebra representations of
$\mathfrak{sl}_2$. We define a second order element of the universal enveloping
algebra $\mathcal{U}(\mathfrak{sl}_2)$ of $\mathfrak{sl}_2(\mathbb{R})$, which,
through the action of a certain infinite dimensional representation of
$\mathfrak{sl}_2(\mathbb{R})$, provides a picture of the asymmetric quantum
Rabi model equivalent to the one drawn by confluent Heun ordinary differential
equations. Using this description, we prove the existence of level crossings in
the spectral graph of the asymmetric quantum Rabi model when the
symmetry-breaking parameter $\epsilon$ is equal to $\frac12$, and conjecture a
formula that ensures likewise the presence of level crossings for general
$\epsilon \in \frac12\mathbb{Z}$. This result on level crossings was
demonstrated numerically by Li and Batchelor in 2015, investigating an earlier
empirical observation by Braak (2011). The first analysis of the degenerate
spectrum was given for the symmetric quantum Rabi model by Ku\'s in 1985. In
our picture, we find a certain reciprocity (or $\mathbb{Z}_2$-symmetry) for
$\epsilon \in \frac12\mathbb{Z}$ if the spectrum is described by
representations of $\frak{sl}_2$. We further discuss briefly the non-degenerate
part of the exceptional spectrum from the viewpoint of infinite dimensional
representations of $\mathfrak{sl}_2(\mathbb{R})$ having lowest weight vectors.
