  We present a novel geometric approach for determining the unique structure of
a Hamiltonian and establishing an instability criterion for quantum quadratic
systems. Our geometric criterion provides insights into the underlying
geometric perspective of instability: A quantum quadratic system is dynamically
unstable if and only if its Hamiltonian is hyperbolic. By applying our
geometric method, we analyze the stability of two-mode and three-mode
optomechanical systems. Remarkably, our approach demonstrates that these
systems can be stabilized over a wider range of system parameters compared to
the conventional rotating wave approximation (RWA) assumption. Furthermore, we
reveal that the systems transit their phases from stable to unstable, when the
system parameters cross specific critical boundaries. The results imply the
presence of multistability in the optomechanical systems.
