  We derive the effective Hamiltonian describing collective oscillations of
Majorana neutrinos with a transition magnetic moment, allowing for the presence
of scalar and pseudoscalar nonstandard neutrino self-interactions (NSSIs).
Using this Hamiltonian, we analyze new flavor instability channels of
collective oscillations in a core-collapse supernova environment that open up
in the presence of a small but nonzero neutrino magnetic moment. It turns out
that, contrary to certain claims in the literature, within the minimally
extended Standard Model (i.e., without NSSIs), no new instabilities arise
within the linear order, nor do they produce any observable signatures in the
neutrino flavor-energy spectra, at least for magnetic moments up to
$10^{-15}~\mu_{\text{B}}$ and quite realistic fields of the order of
$10^{12}~$Gauss. On the other hand, in the presence of NSSIs, new fast and slow
instabilities mixing neutrinos and antineutrinos appear, which show up in the
spectra even for tiny magnetic moments of the order of
$10^{-24}~\mu_{\text{B}}$, leading to considerable distortions of the spectra
and nonstandard spectral splits. We study sensitivity of collective
oscillations to these, NSSI-induced instabilities in detail and discuss the
observability of the NSSI couplings triggering them.
