  The hidden symmetry of the asymmetric quantum Rabi model (AQRM) with a
half-integral bias (ibQRM$_{\ell}$) was uncovered in recent studies by the
explicit construction of operators $J_\ell$ commuting with the Hamiltonian. The
existence of such symmetry has been widely believed to cause the degeneration
of the spectrum, that is, the crossings on the energy curves. In this paper we
propose a conjectural relation between the symmetry and degeneracy for the
ibQRM$_{\ell}$ given explicitly in terms of two polynomials appearing
independently in the respective investigations. Concretely, one of the
polynomials appears as the quotient of the constraint polynomials that assure
the existence of degenerate solutions while the other determines a quadratic
relation (in general, it defines a curve of hyperelliptic type) between the
ibQRM$_{\ell}$ Hamiltonian and its basic commuting operator $J_\ell$. Following
this conjecture, we derive several interesting structural insights of the whole
spectrum. For instance, the energy curves are naturally shown to lie on a
surface determined by the family of hyperelliptic curves by considering the
coupling constant as a variable. This geometric picture contains the
generalization of the parity decomposition of the symmetric quantum Rabi model.
Moreover, it allows us to describe a remarkable approximation of the first
$\ell$ energy curves by the zero-section of the corresponding hyperelliptic
curve. These investigations naturally lead to a geometric picture of the
(hyper-)elliptic surfaces given by the Kodaira-N\'eron type model for a family
of curves over the projective line in connection with the energy curves, which
may be expected to provide a complex analytic proof of the conjecture.
