  We investigate models of charged lepton and neutrino masses and lepton mixing
based on broken modular symmetry. The matter fields in these models are assumed
to transform in irreducible representations of the finite modular group
$\Gamma_4 \simeq S_4$. We analyse the minimal scenario in which the only source
of symmetry breaking is the vacuum expectation value of the modulus field. In
this scenario there is no need to introduce flavon fields. Using the basis for
the lowest weight modular forms found earlier, we build minimal
phenomenologically viable models in which the neutrino masses are generated via
the type I seesaw mechanism. While successfully accommodating charged lepton
masses, neutrino mixing angles and mass-squared differences, these models
predict the values of the lightest neutrino mass (i.e., the absolute neutrino
mass scale), of the Dirac and Majorana CP violation (CPV) phases, as well as
specific correlations between the values of the atmospheric neutrino mixing
parameter $\sin^2\theta_{23}$ and i) the Dirac CPV phase $\delta$, ii) the sum
of the neutrino masses, and iii) the effective Majorana mass in neutrinoless
double beta decay. We consider also the case of residual symmetries
$\mathbb{Z}^{ST}_3$ and $\mathbb{Z}^S_2$ respectively in the charged lepton and
neutrino sectors, corresponding to specific vacuum expectation values of the
modulus.
