  We study a flavor model that the quark sector has the $S_3$ modular
symmetry,while the lepton sector has the $A_4$ modular symmetry. Our model
leads to characteristic quark mass matrices which are consistent with
experimental data of quark masses, mixing angles and the CP violating phase.
The lepton sector is also consistent with the experimental data of neutrino
oscillations. We also study baryon and lepton number violations in our flavor
model.
