  We report on a nonequilibrium quantum phase transition (NQPT) in a hybrid
quantum many-body system consisting of a vibrational mode of a damped
nanomembrane interacting optomechanically with a cavity, whose output light
couples to two internal states of an ultracold Bose gas held in an external
quasi-one-dimensional box potential. For small effective membrane-atom
couplings, the system is in a homogeneous Bose-Einstein condensate (BEC) steady
state, with no membrane displacement. Depending on the transition frequency
between the two internal atomic states, either one or both internal states are
occupied. By increasing the atom-membrane couplings, the system transitions to
a symmetry-broken self-organized BEC phase, which is characterized by a
considerably displaced membrane steady-state and density-wave-like BEC
profiles. This NQPT can be both discontinuous and continuous for a certain
interval of transition frequencies and is purely discontinuous outside of it.
