  The asymmetric quantum Rabi model (AQRM) has a broken $\mathbb{Z}_2$
symmetry, with generally a non-degenerate eigenvalue spectrum. In some special
cases where the asymmetric parameter is a multiple of the cavity frequency,
stable level crossings typical of the $\mathbb{Z}_2$-symmetric quantum Rabi
model are recovered, however, without any obvious parity-like symmetry. This
unknown "symmetry" has thus been referred to as hidden symmetry in the
literature. Here we show that this hidden symmetry is not limited to the AQRM,
but exists in various related light-matter interaction models with an
asymmetric qubit bias term. Conditions under which the hidden symmetry exists
in these models are determined and discussed. By investigating tunnelling
dynamics in the displaced oscillator basis, a strong connection is found
between the hidden symmetry and selective tunnelling.
