  We study a three-mode Hamiltonian modelling a heteronuclear molecular
Bose--Einstein condensate. Two modes are associated with two distinguishable
atomic constituents, which can combine to form a molecule represented by the
third mode. Beginning with a semi-classical analogue of the model, we conduct
an analysis to determine the phase space fixed points of the system.
Bifurcations of the fixed points naturally separate the coupling parameter
space into different regions. Two distinct scenarios are found, dependent on
whether the imbalance between the number operators for the atomic modes is zero
or non-zero. This result suggests the ground-state properties of the model
exhibit an unusual sensitivity on the atomic imbalance. We then test this
finding for the quantum mechanical model. Specifically we use Bethe ansatz
methods, ground-state expectation values, the character of the quantum
dynamics, and ground-state wavefunction overlaps to clarify the nature of the
ground-state phases. The character of the transition is smoothed due to quantum
fluctuations, but we may nonetheless identify the emergence of a quantum phase
boundary in the limit of zero atomic imbalance.
