  The modular $A_4$ symmetry with three moduli is investigated. We assign
different moduli to charged leptons, neutrinos, and quarks. We analyze these
moduli at their fixed points where a residual symmetry exists. We consider two
possibilities for right-handed neutrinos. First, they are assumed to be
singlets under modular symmetry. In this case, we show that the lepton masses
and mixing can be obtained consistently with experimental observations. Second,
they are assigned non-trivially under modular symmetry. We emphasize that a
small deviation from their fixed point is required in this case. Finally, the
quark masses and mixing are generated correctly around the fixed point of their
modulus. In our analysis, we only consider the simple case of weight 2.
