  We study how the degrees of irrationality of moduli spaces of polarized K3
surfaces grow with respect to the genus $g$. We prove that the growth is
bounded by a polynomial function of degree $14+\varepsilon$ for any
$\varepsilon>0$ and, for three sets of infinitely many genera, the bounds can
be refined to polynomials of degree $10$. The main ingredients in our proof are
the modularity of the generating series of Heegner divisors due to Borcherds
and its generalization to higher codimensions due to Kudla, Millson, Zhang,
Bruinier, and Westerholt-Raum. For special genera, the proof is also built upon
the existence of K3 surfaces associated Hodge theoretically with certain cubic
fourfolds, Gushel-Mukai fourfolds, and hyperk\"ahler fourfolds.
