  We explore alternative descriptions of the charged lepton sector in modular
invariant models of lepton masses and mixing angles. In addition to the
modulus, the symmetry breaking sector of our models includes ordinary flavons.
Neutrino mass terms depend only on the modulus and are tailored to minimize the
number of free parameters. The charged lepton Yukawa couplings rely upon the
flavons alone. We build modular invariant models at levels 4 and 5, where
neutrino masses are described both in terms of the Weinberg operator or through
a type I seesaw mechanism. At level 4, our models reproduce the hierarchy among
electron, muon and tau masses by letting the weights play the role of
Froggatt-Nielsen charges. At level 5, our setup allows the treatment of left
and right handed charged leptons on the same footing. We have optimized the
free parameters of our models in order to match the experimental data,
obtaining a good degree of compatibility and predictions for the absolute
neutrino masses and the $CP$ violating phases. At a more fundamental level, the
whole lepton sector could be correctly described by the simultaneous presence
of several moduli. Our examples are meant to make a first step in this
direction.
