  Observations of neutron stars may be used to study aspects of extremely dense
matter, specifically a possibility of phase transitions to exotic states, such
as de-confined quarks.
  We present a novel data analysis method for detecting signatures of
dense-matter phase transitions in sets of mass-radius measurements, and study
its sensitivity with respect to the size of observational errors and the number
of observations. The method is based on machine learning anomaly detection
coupled with normalizing flows technique: the algorithm trained on samples of
astrophysical observations featuring no phase transition signatures interprets
a phase transition sample as an ''anomaly''. For the sake of this study, we
focus on dense-matter equations of state leading to detached branches of
mass-radius sequences (strong phase transitions), use an
astrophysically-informed neutron-star mass function, and various magnitudes of
observational errors and sample sizes.
  The method is shown to reliably detect cases of mass-radius relations with
phase transition signatures, while increasing its sensitivity with decreasing
measurement errors and increasing number of observations. We discuss marginal
cases, when the phase transition mass is located near the edges of the mass
function range. Evaluated on the current state-of-art selection of real
measurements of electromagnetic and gravitational-wave observations, the method
gives inconclusive results, which we interpret as due to small available sample
size, large observational errors and complex systematics.
