  Statistical tasks such as density estimation and approximate Bayesian
inference often involve densities with unknown normalising constants.
Score-based methods, including score matching, are popular techniques as they
are free of normalising constants. Although these methods enjoy theoretical
guarantees, a little-known fact is that they exhibit practical failure modes
when the unnormalised distribution of interest has isolated components -- they
cannot discover isolated components or identify the correct mixing proportions
between components. We demonstrate these findings using simple distributions
and present heuristic attempts to address these issues. We hope to bring the
attention of theoreticians and practitioners to these issues when developing
new algorithms and applications.
