  Let $(X, T)$ be a topological dynamical system. We show that if each
invariant measure of $(X, T)$ gives rise to a measure-theoretic dynamical
system that is either: a. rigid along a sequence of "bounded prime volume" or
b. admits a polynomial rate of rigidity on a linearly dense subset in $C(X)$,
then $(X, T)$ satisfies Sarnak's conjecture on M\"obius disjointness. We show
that the same conclusion also holds if there are countably many invariant
ergodic measures, and each of them satisfies a. or b. This recovers some
earlier results and implies Sarnak's conjecture in the following new cases: for
almost every interval exchange map of $d$ intervals with $d \geq 2$, for
$C^{2+\epsilon}$-smooth skew products over rotations and
$C^{2+\epsilon}$-smooth flows (without fixed points) on the torus. In
particular, these are improvements of earlier results of respectively
Chaika-Eskin, Wang and Huang-Wang-Ye. We also discuss some purely arithmetic
consequences for the Liouville function.
