  The symmetric quantum Rabi model (QRM) is integrable due to a discrete
$\mathbb{Z}_2$-symmetry of the Hamiltonian. This symmetry is generated by a
known involution operator, measuring the parity of the eigenfunctions. An
experimentally relevant modification of the QRM, the asymmetric (or biased)
quantum Rabi model (AQRM) is no longer invariant under this operator, but shows
nevertheless characteristic degeneracies of its spectrum for half-integer
values of $\epsilon$, the parameter governing the asymmetry. In an interesting
recent work (arXiv:2010.02496), an operator has been identified which commutes
with the Hamiltonian $H_{\epsilon}$ of the asymmetric quantum Rabi model for
$\epsilon=\frac{\ell}{2} \, (\ell\in \mathbb{Z})$ and appears to be the
analogue of the parity in the symmetric case. We prove several important
properties of this operator, notably, that it is algebraically independent of
the Hamiltonian $H_{\epsilon}$ and that it essentially generates the commutant
of $H_{\epsilon}$. Then, the expected $\mathbb{Z}_2$-symmetry manifests the
fact that the commuting operator can be captured in the two-fold cover of the
algebra generated by $H_{\epsilon}$, that is, the polynomial ring in
$H_{\epsilon}$.
