  We use a master equation to study the dynamics of two coupled macroscopic
quantum systems (e.g.\ a Josephson junction made of two Bose-Einstein
condensates or two spin states of an ensemble of trapped ions) subject to a
weak continuous measurement. If the coupling between the two systems is
suddenly switched on the resulting dynamics leads to caustics (fold and cusp
catastrophes) in the number-difference probability distribution, and at the
same time the measurement gradually induces a quantum-to-classical transition.
Decoherence is often invoked to help resolve paradoxes associated with
macroscopic quantum mechanics, but here, on the contrary, caustics are
well-behaved in the quantum (many-particle) theory and divergent in the
classical (mean-field) theory. Caustics thus represent a breakdown of the
classical theory towards which decoherence seems to inevitably lead. We find
that measurement backaction plays a crucial role in softening the resulting
singularities and calculate the modification to the Arnol'd index which governs
the scaling of the caustic's amplitude with the number of atoms. The Arnol'd
index acts as a critical exponent for the formation of singularities during
quantum dynamics and its modification by the open nature of the system is
analogous to the modification of the critical exponents of phase transitions
occurring in open systems.
